[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:06:04 dist-test-slave-27sr krb5kdc[22243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:06:04 dist-test-slave-27sr krb5kdc[22243](info): set up 2 sockets
Aug 10 18:06:04 dist-test-slave-27sr krb5kdc[22243](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 18:06:05.757807 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.002s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:06:05 dist-test-slave-27sr krb5kdc[22243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:06:05.903518 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:34843
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:32969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:34843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:06.052136 22257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:06.052659 22257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:06.052714 22257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:06.057717 22257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:06.057819 22257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:06.057842 22257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:06.057862 22257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:06.057880 22257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:06.064680 22257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:32969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:34843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:06.069288 22257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:06.070871 22257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:06.152760 22263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:06.154469 22257 server_base.cc:1034] running on GCE node
W20260810 18:06:06.154534 22266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:06.154832 22264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:06.156528 22257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:06.158722 22257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:06.159992 22257 hybrid_clock.cc:648] HybridClock initialized: now 1786385166159974 us; error 46 us; skew 500 ppm
Aug 10 18:06:06 dist-test-slave-27sr krb5kdc[22243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385166, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:06.163681 22257 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:06:06.165045 22257 webserver.cc:460] Webserver started at http://127.21.182.126:32841/ using document root <none> and password file <none>
I20260810 18:06:06.165874 22257 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:06.165946 22257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:06.166261 22257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:06.169014 22257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "4271265cfa534e23817d26a734ae9ed6"
format_stamp: "Formatted at 2026-08-10 18:06:06 on dist-test-slave-27sr"
server_key: "970eb9b3689699c423fae2efd273e628"
server_key_iv: "e367c3336ead493642d35275d0159ef1"
server_key_version: "encryptionkey@0"
I20260810 18:06:06.169868 22257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "4271265cfa534e23817d26a734ae9ed6"
format_stamp: "Formatted at 2026-08-10 18:06:06 on dist-test-slave-27sr"
server_key: "970eb9b3689699c423fae2efd273e628"
server_key_iv: "e367c3336ead493642d35275d0159ef1"
server_key_version: "encryptionkey@0"
I20260810 18:06:06.176043 22257 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 18:06:06.180274 22274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:06.181922 22257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:06.182204 22257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "4271265cfa534e23817d26a734ae9ed6"
format_stamp: "Formatted at 2026-08-10 18:06:06 on dist-test-slave-27sr"
server_key: "970eb9b3689699c423fae2efd273e628"
server_key_iv: "e367c3336ead493642d35275d0159ef1"
server_key_version: "encryptionkey@0"
I20260810 18:06:06.182418 22257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:06.202848 22257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:06.211292 22257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:06.211503 22257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:06.227198 22257 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:34843
I20260810 18:06:06.227232 22330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:34843 every 8 connection(s)
I20260810 18:06:06.228906 22257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:06.234817 22331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:06.235963 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 22257
I20260810 18:06:06.236174 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:06.238261 22233 external_mini_cluster.cc:1428] Setting key bd24939942bcb3ee09d0c8c5f859cc02
I20260810 18:06:06.246379 22331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6: Bootstrap starting.
I20260810 18:06:06.250949 22331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:06.252079 22331 log.cc:824] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:06.254833 22331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6: No bootstrap required, opened a new log
I20260810 18:06:06.259354 22331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4271265cfa534e23817d26a734ae9ed6" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34843 } }
I20260810 18:06:06.259686 22331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:06.259783 22331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4271265cfa534e23817d26a734ae9ed6, State: Initialized, Role: FOLLOWER
I20260810 18:06:06.261369 22331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4271265cfa534e23817d26a734ae9ed6" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34843 } }
I20260810 18:06:06.261652 22331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:06.261749 22331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:06.261914 22331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:06.263211 22331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4271265cfa534e23817d26a734ae9ed6" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34843 } }
I20260810 18:06:06.263633 22331 leader_election.cc:302] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4271265cfa534e23817d26a734ae9ed6; no voters:
I20260810 18:06:06.264070 22331 leader_election.cc:288] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:06.264436 22336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:06.264777 22336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [term 1 LEADER]: Becoming Leader. State: Replica: 4271265cfa534e23817d26a734ae9ed6, State: Running, Role: LEADER
I20260810 18:06:06.265278 22336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4271265cfa534e23817d26a734ae9ed6" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34843 } }
I20260810 18:06:06.265931 22331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 18:06:06 dist-test-slave-27sr krb5kdc[22243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:06.269821 22337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4271265cfa534e23817d26a734ae9ed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4271265cfa534e23817d26a734ae9ed6" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34843 } } }
I20260810 18:06:06.269970 22337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:06.269925 22338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4271265cfa534e23817d26a734ae9ed6. Latest consensus state: current_term: 1 leader_uuid: "4271265cfa534e23817d26a734ae9ed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4271265cfa534e23817d26a734ae9ed6" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34843 } } }
I20260810 18:06:06.270045 22338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6 [sys.catalog]: This master's current role is: LEADER
W20260810 18:06:06.275168 22353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:06:06.275332 22353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:06:06.275486 22354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:06.275836 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:06.248368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.21.182.126:34843)
0810 18:06:06.248974 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:06.248989 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:06.249048 (+ 59us) server_negotiation.cc:407] Connection header received
0810 18:06:06.254092 (+ 5044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:06.254112 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:06.254573 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:06.255213 (+ 640us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:06.258170 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:06.259799 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:06.261475 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:06.262344 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:06.262599 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:06.267957 (+ 5358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:06.267968 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:06.267991 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:06.268038 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:06.270991 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:06.271665 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:06.271673 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:06.271680 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:06.271815 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:06.272338 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:06.272343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:06.272345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:06.273001 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:06.273362 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:06.274103 (+ 741us) server_negotiation.cc:299] Negotiation successful
0810 18:06:06.274447 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":187,"threads_started":1}
I20260810 18:06:06.276794 22354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:06.286056 22354 catalog_manager.cc:1347] Generated new cluster ID: a33e936fc5d44c47842c85f4562a42f6
I20260810 18:06:06.286170 22354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:06.340241 22354 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:06.341957 22354 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:06.370059 22354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4271265cfa534e23817d26a734ae9ed6: Generated new TSK 0
I20260810 18:06:06.371145 22354 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:06:06.420308 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:34843
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:32969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:06.592799 22361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:06.593119 22361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:06.593173 22361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:06.598142 22361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:06.598239 22361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:06.598307 22361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:06.604193 22361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:32969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:34843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:06.605521 22361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:06.606822 22361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:06.617835 22367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:06.617939 22369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:06.617939 22366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:06.618455 22361 server_base.cc:1034] running on GCE node
I20260810 18:06:06.618903 22361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:06.619670 22361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:06.620869 22361 hybrid_clock.cc:648] HybridClock initialized: now 1786385166620858 us; error 60 us; skew 500 ppm
Aug 10 18:06:06 dist-test-slave-27sr krb5kdc[22243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385166, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:06.624279 22361 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:06:06.625509 22361 webserver.cc:460] Webserver started at http://127.21.182.65:36757/ using document root <none> and password file <none>
I20260810 18:06:06.626194 22361 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:06.626312 22361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:06.626588 22361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:06.629135 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "54b9ad50d9e5459fbb31a8692277be4d"
format_stamp: "Formatted at 2026-08-10 18:06:06 on dist-test-slave-27sr"
server_key: "de1106eac51e07f76af6058de1d2497a"
server_key_iv: "d23255cfdd84382e1ed2d11c1e5f4fe3"
server_key_version: "encryptionkey@0"
I20260810 18:06:06.629832 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "54b9ad50d9e5459fbb31a8692277be4d"
format_stamp: "Formatted at 2026-08-10 18:06:06 on dist-test-slave-27sr"
server_key: "de1106eac51e07f76af6058de1d2497a"
server_key_iv: "d23255cfdd84382e1ed2d11c1e5f4fe3"
server_key_version: "encryptionkey@0"
I20260810 18:06:06.634279 22361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:06:06.637208 22379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:06.638536 22361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:06:06.638706 22361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "54b9ad50d9e5459fbb31a8692277be4d"
format_stamp: "Formatted at 2026-08-10 18:06:06 on dist-test-slave-27sr"
server_key: "de1106eac51e07f76af6058de1d2497a"
server_key_iv: "d23255cfdd84382e1ed2d11c1e5f4fe3"
server_key_version: "encryptionkey@0"
I20260810 18:06:06.638846 22361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:06.674685 22361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:06.678584 22361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:06.678964 22361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:06.679817 22361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:06.680956 22361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:06.681035 22361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:06.681111 22361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:06.681164 22361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:06.694972 22361 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:42637
I20260810 18:06:06.695037 22513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:42637 every 8 connection(s)
I20260810 18:06:06.696118 22361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:06.701011 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 22361
I20260810 18:06:06.701122 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:06:06.701455 22233 external_mini_cluster.cc:1428] Setting key f43b2cc0ef342ddd40dc2fa7cbf86350
Aug 10 18:06:06 dist-test-slave-27sr krb5kdc[22243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385166, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
I20260810 18:06:06.714212 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:06.698691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:38693 (local address 127.21.182.126:34843)
0810 18:06:06.698886 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:06.698890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:06.700319 (+ 1429us) server_negotiation.cc:407] Connection header received
0810 18:06:06.701452 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:06.701456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:06.701520 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:06.701609 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:06.703336 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:06.703961 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:06.706213 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:06.706636 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:06.706772 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:06.710178 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:06.710183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:06.710185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:06.710213 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:06.711833 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:06.712505 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:06.712509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:06.712511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:06.712603 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:06.712990 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:06.712995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:06.712997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:06.713207 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:06.713326 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:06.713884 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 18:06:06.714034 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 18:06:06.715245 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:06.699192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:34843 (local address 127.21.182.65:38693)
0810 18:06:06.700120 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:06.700197 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:06.701215 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:06.701875 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:06.701885 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:06.702475 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:06.703143 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:06.703170 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:06.704102 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:06.704109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:06.706033 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:06.706051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:06.706909 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:06.706912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:06.707085 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:06.707851 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:06.707890 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:06.709974 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:06.712056 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:06.712064 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:06.712092 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:06.712382 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:06.712702 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:06.712706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:06.712708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:06.712882 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:06.713320 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:06.713328 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:06.713617 (+ 289us) client_negotiation.cc:769] Sending connection context
0810 18:06:06.713832 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 18:06:06.714204 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":206,"threads_started":1}
I20260810 18:06:06.716828 22514 heartbeater.cc:347] Connected to a master server at 127.21.182.126:34843
I20260810 18:06:06.717094 22514 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
I20260810 18:06:06.717707 22514 heartbeater.cc:511] Master 127.21.182.126:34843 requested a full tablet report, sending...
I20260810 18:06:06.719851 22293 ts_manager.cc:194] Registered new tserver with Master: 54b9ad50d9e5459fbb31a8692277be4d (127.21.182.65:42637)
I20260810 18:06:06.721946 22293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:38693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:06:06.735591 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:34843
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:32969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:06.916132 22519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:06.916430 22519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:06.916477 22519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:06.920817 22519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:06.920897 22519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:06.920960 22519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:06.927752 22519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:32969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:34843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:06.929136 22519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:06.930373 22519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:06.944507 22530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:06.944553 22526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:06.944526 22527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:06.944815 22519 server_base.cc:1034] running on GCE node
I20260810 18:06:06.945377 22519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:06.946204 22519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:06.947440 22519 hybrid_clock.cc:648] HybridClock initialized: now 1786385166947406 us; error 49 us; skew 500 ppm
Aug 10 18:06:06 dist-test-slave-27sr krb5kdc[22243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385166, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:06.951570 22519 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:06:06.952870 22519 webserver.cc:460] Webserver started at http://127.21.182.66:42207/ using document root <none> and password file <none>
I20260810 18:06:06.953600 22519 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:06.953712 22519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:06.953990 22519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:06.956840 22519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "91aa4052b4b0471692196955a3afd77c"
format_stamp: "Formatted at 2026-08-10 18:06:06 on dist-test-slave-27sr"
server_key: "d5cab9ac331b49eff4f42aa1fede565c"
server_key_iv: "4b00fcf55fd12b5b4c82b0b889f27f37"
server_key_version: "encryptionkey@0"
I20260810 18:06:06.957612 22519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "91aa4052b4b0471692196955a3afd77c"
format_stamp: "Formatted at 2026-08-10 18:06:06 on dist-test-slave-27sr"
server_key: "d5cab9ac331b49eff4f42aa1fede565c"
server_key_iv: "4b00fcf55fd12b5b4c82b0b889f27f37"
server_key_version: "encryptionkey@0"
I20260810 18:06:06.962364 22519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 18:06:06.965415 22539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:06.967001 22519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:06:06.967238 22519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "91aa4052b4b0471692196955a3afd77c"
format_stamp: "Formatted at 2026-08-10 18:06:06 on dist-test-slave-27sr"
server_key: "d5cab9ac331b49eff4f42aa1fede565c"
server_key_iv: "4b00fcf55fd12b5b4c82b0b889f27f37"
server_key_version: "encryptionkey@0"
I20260810 18:06:06.967375 22519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:06.994580 22519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:06.998421 22519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:06.998688 22519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:06.999537 22519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:07.000615 22519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:07.000670 22519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:07.000715 22519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:07.000769 22519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:07.019016 22519 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:45589
I20260810 18:06:07.019033 22672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:45589 every 8 connection(s)
I20260810 18:06:07.020304 22519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:06:07.027706 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 22519
I20260810 18:06:07.027861 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:06:07.028172 22233 external_mini_cluster.cc:1428] Setting key ffe09386193163c5dede008bd4f47c76
Aug 10 18:06:07 dist-test-slave-27sr krb5kdc[22243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385166, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:07.038779 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:07.022738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:42201 (local address 127.21.182.126:34843)
0810 18:06:07.022895 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:07.022899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:07.025447 (+ 2548us) server_negotiation.cc:407] Connection header received
0810 18:06:07.026626 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:07.026630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:07.026713 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:07.026825 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:07.028355 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:07.029117 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:07.030704 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:07.031297 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:07.031447 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:07.034618 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:07.034622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:07.034625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:07.034652 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:07.036449 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:07.037077 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:07.037080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:07.037082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:07.037162 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:07.037534 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:07.037537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:07.037539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:07.037760 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:07.037873 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:07.038433 (+ 560us) server_negotiation.cc:299] Negotiation successful
0810 18:06:07.038573 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:06:07.039688 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:07.023092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:34843 (local address 127.21.182.66:42201)
0810 18:06:07.025256 (+ 2164us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:07.025308 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:07.026375 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:07.027026 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:07.027034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:07.027488 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:07.028166 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:07.028190 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:07.029291 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:07.029294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:07.030558 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:07.030570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:07.031485 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:07.031488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:07.031701 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:07.032611 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:07.032629 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:07.034373 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:07.036593 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:07.036601 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:07.036620 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:07.036946 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:07.037264 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:07.037267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:07.037269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:07.037427 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:07.037874 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:07.037885 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:07.038166 (+ 281us) client_negotiation.cc:769] Sending connection context
0810 18:06:07.038372 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 18:06:07.038758 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1930,"thread_start_us":1710,"threads_started":1}
I20260810 18:06:07.041114 22673 heartbeater.cc:347] Connected to a master server at 127.21.182.126:34843
I20260810 18:06:07.041388 22673 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
I20260810 18:06:07.041977 22673 heartbeater.cc:511] Master 127.21.182.126:34843 requested a full tablet report, sending...
I20260810 18:06:07.043404 22293 ts_manager.cc:194] Registered new tserver with Master: 91aa4052b4b0471692196955a3afd77c (127.21.182.66:45589)
I20260810 18:06:07.044165 22293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:42201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:06:07.068272 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:34843
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:32969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:07.217136 22679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:07.217429 22679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:07.217528 22679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:07.221884 22679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:07.221961 22679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:07.222060 22679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:07.228782 22679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:32969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:34843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:07.230017 22679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:07.231230 22679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:07.243166 22686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:07.243119 22688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:07.243098 22685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:07.243613 22679 server_base.cc:1034] running on GCE node
I20260810 18:06:07.244171 22679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:07.244853 22679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:07.246038 22679 hybrid_clock.cc:648] HybridClock initialized: now 1786385167246017 us; error 38 us; skew 500 ppm
Aug 10 18:06:07 dist-test-slave-27sr krb5kdc[22243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385167, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:07.249600 22679 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:06:07.250535 22679 webserver.cc:460] Webserver started at http://127.21.182.67:34895/ using document root <none> and password file <none>
I20260810 18:06:07.251116 22679 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:07.251178 22679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:07.251354 22679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:07.253844 22679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "939f3340e10c48e9965fd83c45ea8d5d"
format_stamp: "Formatted at 2026-08-10 18:06:07 on dist-test-slave-27sr"
server_key: "14bd19b69cbcb9e764c44faa613bfd65"
server_key_iv: "55df45dc27bd34500d69d169029f3e2e"
server_key_version: "encryptionkey@0"
I20260810 18:06:07.254514 22679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "939f3340e10c48e9965fd83c45ea8d5d"
format_stamp: "Formatted at 2026-08-10 18:06:07 on dist-test-slave-27sr"
server_key: "14bd19b69cbcb9e764c44faa613bfd65"
server_key_iv: "55df45dc27bd34500d69d169029f3e2e"
server_key_version: "encryptionkey@0"
I20260810 18:06:07.259312 22679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 18:06:07.262627 22695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:07.264328 22679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:07.264473 22679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "939f3340e10c48e9965fd83c45ea8d5d"
format_stamp: "Formatted at 2026-08-10 18:06:07 on dist-test-slave-27sr"
server_key: "14bd19b69cbcb9e764c44faa613bfd65"
server_key_iv: "55df45dc27bd34500d69d169029f3e2e"
server_key_version: "encryptionkey@0"
I20260810 18:06:07.264593 22679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:07.306594 22679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:07.310670 22679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:07.310916 22679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:07.311694 22679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:07.312767 22679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:07.312852 22679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:07.312947 22679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:07.313002 22679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:07.336423 22679 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:34189
I20260810 18:06:07.336515 22819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:34189 every 8 connection(s)
I20260810 18:06:07.337615 22679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:06:07.338555 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 22679
I20260810 18:06:07.338699 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:06:07.338999 22233 external_mini_cluster.cc:1428] Setting key 3e97339cb69693cd4eee65804b11d74f
Aug 10 18:06:07 dist-test-slave-27sr krb5kdc[22243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385167, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:07.353986 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:07.340365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:50581 (local address 127.21.182.126:34843)
0810 18:06:07.340526 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:07.340531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:07.342303 (+ 1772us) server_negotiation.cc:407] Connection header received
0810 18:06:07.343549 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:07.343551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:07.343620 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:07.343698 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:07.345245 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:07.345828 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:07.347136 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:07.347421 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:07.347516 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:07.350402 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:07.350406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:07.350408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:07.350434 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:07.351754 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:07.352313 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:07.352316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:07.352317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:07.352395 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:07.352707 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:07.352710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:07.352711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:07.352947 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:07.353046 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:07.353676 (+ 630us) server_negotiation.cc:299] Negotiation successful
0810 18:06:07.353830 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:06:07.355177 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:07.340989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:34843 (local address 127.21.182.67:50581)
0810 18:06:07.342127 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:07.342193 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:07.343343 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:07.343847 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:07.343854 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:07.344475 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:07.345098 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:07.345114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:07.345946 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:07.345948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:07.347000 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:07.347010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:07.347570 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:07.347573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:07.347782 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:07.348573 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:07.348591 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:07.350240 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:07.351882 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:07.351887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:07.351944 (+ 57us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:07.352223 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:07.352480 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:07.352482 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:07.352483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:07.352622 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:07.353049 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:07.353054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:07.353420 (+ 366us) client_negotiation.cc:769] Sending connection context
0810 18:06:07.353584 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 18:06:07.353980 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"thread_start_us":316,"threads_started":1}
I20260810 18:06:07.356801 22820 heartbeater.cc:347] Connected to a master server at 127.21.182.126:34843
I20260810 18:06:07.357087 22820 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:07.357678 22820 heartbeater.cc:511] Master 127.21.182.126:34843 requested a full tablet report, sending...
I20260810 18:06:07.358899 22293 ts_manager.cc:194] Registered new tserver with Master: 939f3340e10c48e9965fd83c45ea8d5d (127.21.182.67:34189)
I20260810 18:06:07.359537 22293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:50581
I20260810 18:06:07.367165 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:06:07 dist-test-slave-27sr krb5kdc[22243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385167, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:06:07 dist-test-slave-27sr krb5kdc[22243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385167, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.182.65@KRBTEST.COM
I20260810 18:06:07.399943 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:07.389470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.21.182.65:42637)
0810 18:06:07.389882 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:07.389890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:07.389919 (+ 29us) server_negotiation.cc:407] Connection header received
0810 18:06:07.390060 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:07.390066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:07.390269 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:07.390407 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:07.391281 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:07.392189 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:07.393126 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:07.393513 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:07.393602 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:07.396346 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:07.396356 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:07.396371 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:07.396396 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:07.398132 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:07.398533 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:07.398538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:07.398539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:07.398622 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:07.398931 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:07.398933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:07.398934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:07.399177 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:07.399342 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:07.399528 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 18:06:07.399698 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":124,"threads_started":1}
W20260810 18:06:07.401062 22403 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48506
Aug 10 18:06:07 dist-test-slave-27sr krb5kdc[22243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:06:07 dist-test-slave-27sr krb5kdc[22243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.65@KRBTEST.COM
I20260810 18:06:07.435066 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:07.425500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48514 (local address 127.21.182.65:42637)
0810 18:06:07.425793 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:07.425799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:07.425991 (+ 192us) server_negotiation.cc:407] Connection header received
0810 18:06:07.426193 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:07.426198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:07.426259 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:07.426350 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:07.427089 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:07.427597 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:07.428554 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:07.428793 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:07.428908 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:07.431710 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:07.431714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:07.431716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:07.431752 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:07.433221 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:07.433690 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:07.433694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:07.433697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:07.433796 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:07.434104 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:07.434108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:07.434110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:07.434306 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:07.434464 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:07.434722 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 18:06:07.434871 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260810 18:06:07.436426 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 22361
I20260810 18:06:07.446113 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 22519
I20260810 18:06:07.456311 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 22679
I20260810 18:06:07.467731 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 22257
2026-08-10T18:06:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2921 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:06:07 dist-test-slave-27sr krb5kdc[22835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:06:07 dist-test-slave-27sr krb5kdc[22835](info): set up 2 sockets
Aug 10 18:06:07 dist-test-slave-27sr krb5kdc[22835](info): commencing operation
krb5kdc: starting...
W20260810 18:06:08.624475 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.009s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:06:08 dist-test-slave-27sr krb5kdc[22835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:06:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:06:08.767805 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:33241
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:33241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:08.888033 22850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:08.888314 22850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:08.888355 22850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:08.892445 22850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:08.892508 22850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:08.892526 22850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:08.892544 22850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:08.892560 22850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:08.897984 22850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:33241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:33241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:08.899107 22850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:08.900283 22850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:08.909444 22855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:08.909440 22858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:08.909435 22856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:08.909924 22850 server_base.cc:1034] running on GCE node
I20260810 18:06:08.910640 22850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:08.911847 22850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:08.913084 22850 hybrid_clock.cc:648] HybridClock initialized: now 1786385168913061 us; error 32 us; skew 500 ppm
Aug 10 18:06:08 dist-test-slave-27sr krb5kdc[22835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385168, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:08.916174 22850 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:06:08.917099 22850 webserver.cc:460] Webserver started at http://127.21.182.126:35787/ using document root <none> and password file <none>
I20260810 18:06:08.917711 22850 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:08.917807 22850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:08.918052 22850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:08.920226 22850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "4e190aa819ba45cabf96f1a19889d927"
format_stamp: "Formatted at 2026-08-10 18:06:08 on dist-test-slave-27sr"
server_key: "a942173cf1a6081695b3ff9198176cf9"
server_key_iv: "060f7713c46201ee5b2662e312284a2f"
server_key_version: "encryptionkey@0"
I20260810 18:06:08.920806 22850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "4e190aa819ba45cabf96f1a19889d927"
format_stamp: "Formatted at 2026-08-10 18:06:08 on dist-test-slave-27sr"
server_key: "a942173cf1a6081695b3ff9198176cf9"
server_key_iv: "060f7713c46201ee5b2662e312284a2f"
server_key_version: "encryptionkey@0"
I20260810 18:06:08.924849 22850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 18:06:08.927443 22866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:08.928965 22850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:06:08.929142 22850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "4e190aa819ba45cabf96f1a19889d927"
format_stamp: "Formatted at 2026-08-10 18:06:08 on dist-test-slave-27sr"
server_key: "a942173cf1a6081695b3ff9198176cf9"
server_key_iv: "060f7713c46201ee5b2662e312284a2f"
server_key_version: "encryptionkey@0"
I20260810 18:06:08.929284 22850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:08.977991 22850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:08.981725 22850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:08.981969 22850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:08.990296 22850 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:33241
I20260810 18:06:08.990343 22925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:33241 every 8 connection(s)
I20260810 18:06:08.991408 22850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:08.994477 22927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:08.996120 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 22850
I20260810 18:06:08.996243 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:08.996554 22233 external_mini_cluster.cc:1428] Setting key 83683d16db8c223cbf99d5bbb23d46d3
I20260810 18:06:09.001179 22927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927: Bootstrap starting.
I20260810 18:06:09.003575 22927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:09.004490 22927 log.cc:824] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:06:09 dist-test-slave-27sr krb5kdc[22835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:09.006705 22927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927: No bootstrap required, opened a new log
I20260810 18:06:09.010645 22927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e190aa819ba45cabf96f1a19889d927" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33241 } }
I20260810 18:06:09.010897 22927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:09.010958 22927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e190aa819ba45cabf96f1a19889d927, State: Initialized, Role: FOLLOWER
I20260810 18:06:09.011439 22927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e190aa819ba45cabf96f1a19889d927" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33241 } }
I20260810 18:06:09.011523 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:08.998233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.21.182.126:33241)
0810 18:06:08.998713 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:08.998730 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:08.998770 (+ 40us) server_negotiation.cc:407] Connection header received
0810 18:06:08.999452 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:08.999471 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:08.999810 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:09.000240 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:09.001221 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.002270 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.003176 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.003695 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.003871 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:09.006717 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:09.006722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:09.006736 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:09.006788 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:09.008579 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:09.009039 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:09.009044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:09.009049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:09.009160 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:09.009453 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:09.009457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:09.009458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:09.009875 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:09.010093 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:09.010362 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 18:06:09.010655 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":126,"threads_started":1}
I20260810 18:06:09.011569 22927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:09.011649 22927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:09.011741 22927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:09.012787 22927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e190aa819ba45cabf96f1a19889d927" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33241 } }
I20260810 18:06:09.013134 22927 leader_election.cc:302] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e190aa819ba45cabf96f1a19889d927; no voters:
I20260810 18:06:09.013484 22927 leader_election.cc:288] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:09.013577 22932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:09.013768 22932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [term 1 LEADER]: Becoming Leader. State: Replica: 4e190aa819ba45cabf96f1a19889d927, State: Running, Role: LEADER
I20260810 18:06:09.014087 22932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e190aa819ba45cabf96f1a19889d927" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33241 } }
I20260810 18:06:09.014799 22927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:09.015321 22932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e190aa819ba45cabf96f1a19889d927. Latest consensus state: current_term: 1 leader_uuid: "4e190aa819ba45cabf96f1a19889d927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e190aa819ba45cabf96f1a19889d927" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33241 } } }
I20260810 18:06:09.015439 22932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:09.015748 22934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e190aa819ba45cabf96f1a19889d927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e190aa819ba45cabf96f1a19889d927" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33241 } } }
I20260810 18:06:09.015867 22934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:09.015878 22937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:09.019238 22937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:09.025295 22937 catalog_manager.cc:1347] Generated new cluster ID: 5ee42dc6a901452d9f6985e949b8818b
I20260810 18:06:09.025408 22937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:09.040267 22937 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:09.041332 22937 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:09.046808 22937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e190aa819ba45cabf96f1a19889d927: Generated new TSK 0
I20260810 18:06:09.047582 22937 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:06:09.088778 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33241
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:09.212452 22956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:09.212749 22956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:09.212796 22956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:09.217383 22956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:09.217456 22956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:09.217521 22956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:09.223086 22956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33241
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:09.224261 22956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:09.225273 22956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:09.236130 22965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:09.236189 22956 server_base.cc:1034] running on GCE node
W20260810 18:06:09.236133 22961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:09.236490 22963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:09.236955 22956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:09.237599 22956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:09.238773 22956 hybrid_clock.cc:648] HybridClock initialized: now 1786385169238755 us; error 31 us; skew 500 ppm
Aug 10 18:06:09 dist-test-slave-27sr krb5kdc[22835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385169, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:09.241992 22956 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:06:09.243036 22956 webserver.cc:460] Webserver started at http://127.21.182.65:36983/ using document root <none> and password file <none>
I20260810 18:06:09.243697 22956 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:09.243765 22956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:09.244071 22956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:09.246304 22956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "ea99622d19b7474687cec6ea8e714a89"
format_stamp: "Formatted at 2026-08-10 18:06:09 on dist-test-slave-27sr"
server_key: "bdca3ba5af7c6d4191cb89fac6021228"
server_key_iv: "efebb924d3421e6ecf9d499ffda33594"
server_key_version: "encryptionkey@0"
I20260810 18:06:09.246980 22956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "ea99622d19b7474687cec6ea8e714a89"
format_stamp: "Formatted at 2026-08-10 18:06:09 on dist-test-slave-27sr"
server_key: "bdca3ba5af7c6d4191cb89fac6021228"
server_key_iv: "efebb924d3421e6ecf9d499ffda33594"
server_key_version: "encryptionkey@0"
I20260810 18:06:09.251309 22956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260810 18:06:09.254180 22972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:09.255604 22956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:09.255837 22956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "ea99622d19b7474687cec6ea8e714a89"
format_stamp: "Formatted at 2026-08-10 18:06:09 on dist-test-slave-27sr"
server_key: "bdca3ba5af7c6d4191cb89fac6021228"
server_key_iv: "efebb924d3421e6ecf9d499ffda33594"
server_key_version: "encryptionkey@0"
I20260810 18:06:09.256034 22956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:09.320304 22956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:09.323729 22956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:09.323998 22956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:09.324683 22956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:09.325604 22956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:09.325680 22956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:09.325765 22956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:09.325809 22956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:09.336030 22956 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:42367
I20260810 18:06:09.336050 23103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:42367 every 8 connection(s)
I20260810 18:06:09.337121 22956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:09.338187 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 22956
I20260810 18:06:09.338305 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:06:09.338598 22233 external_mini_cluster.cc:1428] Setting key 97e0118f8556476bbbe1a3d0ec283802
Aug 10 18:06:09 dist-test-slave-27sr krb5kdc[22835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385169, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
I20260810 18:06:09.354178 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:09.339440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:54873 (local address 127.21.182.126:33241)
0810 18:06:09.339586 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:09.339591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:09.340850 (+ 1259us) server_negotiation.cc:407] Connection header received
0810 18:06:09.342340 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:09.342343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:09.342398 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:09.342488 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:09.343924 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.344600 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.346099 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.346596 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.346753 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:09.349935 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:09.349938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:09.349940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:09.349969 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:09.351607 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:09.352312 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:09.352316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:09.352320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:09.352412 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:09.352914 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:09.352917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:09.352918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:09.353126 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:09.353252 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:09.353848 (+ 596us) server_negotiation.cc:299] Negotiation successful
0810 18:06:09.354003 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
I20260810 18:06:09.355382 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:09.340021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33241 (local address 127.21.182.65:54873)
0810 18:06:09.340650 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:09.340714 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:09.342027 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:09.342749 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:09.342758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:09.343161 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:09.343693 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:09.343707 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.344919 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.344922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:09.345936 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:09.345947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.346841 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.346843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:09.346999 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:09.347746 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:09.347779 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:09.349684 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:09.351755 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:09.351762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:09.351787 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:09.352160 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:09.352512 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:09.352515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:09.352517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:09.352786 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:09.353243 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:09.353251 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:09.353557 (+ 306us) client_negotiation.cc:769] Sending connection context
0810 18:06:09.353783 (+ 226us) client_negotiation.cc:240] Negotiation successful
0810 18:06:09.354175 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":169,"threads_started":1}
I20260810 18:06:09.357120 23104 heartbeater.cc:347] Connected to a master server at 127.21.182.126:33241
I20260810 18:06:09.357517 23104 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:09.358280 23104 heartbeater.cc:511] Master 127.21.182.126:33241 requested a full tablet report, sending...
I20260810 18:06:09.360793 22884 ts_manager.cc:194] Registered new tserver with Master: ea99622d19b7474687cec6ea8e714a89 (127.21.182.65:42367)
I20260810 18:06:09.362120 22884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:54873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:06:09.375826 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33241
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:09.497385 23109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:09.497689 23109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:09.497740 23109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:09.501845 23109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:09.501915 23109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:09.501976 23109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:09.507411 23109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33241
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:09.508577 23109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:09.509590 23109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:09.519726 23116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:09.519791 23117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:09.519721 23120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:09.520048 23109 server_base.cc:1034] running on GCE node
I20260810 18:06:09.520486 23109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:09.521122 23109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:09.522297 23109 hybrid_clock.cc:648] HybridClock initialized: now 1786385169522284 us; error 33 us; skew 500 ppm
Aug 10 18:06:09 dist-test-slave-27sr krb5kdc[22835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385169, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:09.525213 23109 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:06:09.526085 23109 webserver.cc:460] Webserver started at http://127.21.182.66:32809/ using document root <none> and password file <none>
I20260810 18:06:09.526659 23109 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:09.526750 23109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:09.527056 23109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:09.529179 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "b4b1cc5320bd41c4850fe59c51c010f2"
format_stamp: "Formatted at 2026-08-10 18:06:09 on dist-test-slave-27sr"
server_key: "56550b8345928b7bd49ee5f3a7298213"
server_key_iv: "a18460d3764f5349f670ada555b2f7b2"
server_key_version: "encryptionkey@0"
I20260810 18:06:09.529757 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "b4b1cc5320bd41c4850fe59c51c010f2"
format_stamp: "Formatted at 2026-08-10 18:06:09 on dist-test-slave-27sr"
server_key: "56550b8345928b7bd49ee5f3a7298213"
server_key_iv: "a18460d3764f5349f670ada555b2f7b2"
server_key_version: "encryptionkey@0"
I20260810 18:06:09.533547 23109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:06:09.536048 23127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:09.537276 23109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:06:09.537410 23109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "b4b1cc5320bd41c4850fe59c51c010f2"
format_stamp: "Formatted at 2026-08-10 18:06:09 on dist-test-slave-27sr"
server_key: "56550b8345928b7bd49ee5f3a7298213"
server_key_iv: "a18460d3764f5349f670ada555b2f7b2"
server_key_version: "encryptionkey@0"
I20260810 18:06:09.537519 23109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:09.549685 23109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:09.553023 23109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:09.553262 23109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:09.553930 23109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:09.554877 23109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:09.554967 23109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:09.555070 23109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:09.555126 23109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:09.565937 23109 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:34823
I20260810 18:06:09.565958 23262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:34823 every 8 connection(s)
I20260810 18:06:09.566967 23109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:06:09.574012 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 23109
I20260810 18:06:09.574105 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:06:09.574389 22233 external_mini_cluster.cc:1428] Setting key 7c7f21a96fb8a151feb4cfd98d03a839
Aug 10 18:06:09 dist-test-slave-27sr krb5kdc[22835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385169, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:09.581387 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:09.569067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:34065 (local address 127.21.182.126:33241)
0810 18:06:09.569317 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:09.569322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:09.570433 (+ 1111us) server_negotiation.cc:407] Connection header received
0810 18:06:09.571230 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:09.571234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:09.571303 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:09.571390 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:09.572636 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.573210 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.574435 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.574720 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.574845 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:09.577378 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:09.577382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:09.577385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:09.577418 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:09.579110 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:09.579669 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:09.579675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:09.579678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:09.579769 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:09.580195 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:09.580199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:09.580203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:09.580432 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:09.580549 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:09.581080 (+ 531us) server_negotiation.cc:299] Negotiation successful
0810 18:06:09.581211 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 18:06:09.582216 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:09.569543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33241 (local address 127.21.182.66:34065)
0810 18:06:09.570191 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:09.570234 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:09.571014 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:09.571557 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:09.571564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:09.571991 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:09.572480 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:09.572493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.573347 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.573349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:09.574308 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:09.574318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.574883 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.574887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:09.575058 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:09.575686 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:09.575703 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:09.577210 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:09.579244 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:09.579251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:09.579267 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:09.579555 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:09.579877 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:09.579882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:09.579884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:09.580062 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:09.580546 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:09.580554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:09.580824 (+ 270us) client_negotiation.cc:769] Sending connection context
0810 18:06:09.581018 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 18:06:09.581360 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":289,"threads_started":1}
I20260810 18:06:09.583472 23263 heartbeater.cc:347] Connected to a master server at 127.21.182.126:33241
I20260810 18:06:09.583734 23263 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:09.584298 23263 heartbeater.cc:511] Master 127.21.182.126:33241 requested a full tablet report, sending...
I20260810 18:06:09.585415 22884 ts_manager.cc:194] Registered new tserver with Master: b4b1cc5320bd41c4850fe59c51c010f2 (127.21.182.66:34823)
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
I20260810 18:06:09.586143 22884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:34065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:06:09.607319 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33241
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:09.731777 23269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:09.732168 23269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:09.732254 23269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:09.736984 23269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:09.737093 23269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:09.737195 23269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:09.743414 23269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33241
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:09.744800 23269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:09.745909 23269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:09.758118 23279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:09.758235 23277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:09.758414 23276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:09.758883 23269 server_base.cc:1034] running on GCE node
I20260810 18:06:09.759547 23269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:09.760823 23269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:09.762018 23269 hybrid_clock.cc:648] HybridClock initialized: now 1786385169761984 us; error 52 us; skew 500 ppm
Aug 10 18:06:09 dist-test-slave-27sr krb5kdc[22835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385169, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:09.766711 23269 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:06:09.768237 23269 webserver.cc:460] Webserver started at http://127.21.182.67:37245/ using document root <none> and password file <none>
I20260810 18:06:09.769142 23269 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:09.769259 23269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:09.769541 23269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:09.772877 23269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "5bfb8b68c2d84b609636263aae0773bf"
format_stamp: "Formatted at 2026-08-10 18:06:09 on dist-test-slave-27sr"
server_key: "023dbcb8a71f13531ed7339dcc286675"
server_key_iv: "938690bcc56e04e3c39869649f93834b"
server_key_version: "encryptionkey@0"
I20260810 18:06:09.773743 23269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "5bfb8b68c2d84b609636263aae0773bf"
format_stamp: "Formatted at 2026-08-10 18:06:09 on dist-test-slave-27sr"
server_key: "023dbcb8a71f13531ed7339dcc286675"
server_key_iv: "938690bcc56e04e3c39869649f93834b"
server_key_version: "encryptionkey@0"
I20260810 18:06:09.779717 23269 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260810 18:06:09.784051 23287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:09.789935 23269 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260810 18:06:09.790107 23269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "5bfb8b68c2d84b609636263aae0773bf"
format_stamp: "Formatted at 2026-08-10 18:06:09 on dist-test-slave-27sr"
server_key: "023dbcb8a71f13531ed7339dcc286675"
server_key_iv: "938690bcc56e04e3c39869649f93834b"
server_key_version: "encryptionkey@0"
I20260810 18:06:09.790264 23269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:09.809556 23269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:09.814718 23269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:09.814970 23269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:09.815886 23269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:09.817085 23269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:09.817139 23269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:09.817183 23269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:09.817247 23269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:09.828774 23269 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:35755
I20260810 18:06:09.828868 23421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:35755 every 8 connection(s)
I20260810 18:06:09.829903 23269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:06:09.836935 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 23269
I20260810 18:06:09.837108 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:06:09.837404 22233 external_mini_cluster.cc:1428] Setting key 281796928d35397934fd19b7e6024c5f
Aug 10 18:06:09 dist-test-slave-27sr krb5kdc[22835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385169, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:09.846530 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:09.832257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:40429 (local address 127.21.182.126:33241)
0810 18:06:09.832432 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:09.832437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:09.833732 (+ 1295us) server_negotiation.cc:407] Connection header received
0810 18:06:09.835170 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:09.835173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:09.835235 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:09.835314 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:09.836895 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.837577 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.839116 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.839352 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.839432 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:09.842635 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:09.842638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:09.842640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:09.842670 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:09.844346 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:09.844933 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:09.844936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:09.844937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:09.845021 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:09.845393 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:09.845395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:09.845396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:09.845596 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:09.845673 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:09.846268 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 18:06:09.846401 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 18:06:09.847533 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:09.832646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33241 (local address 127.21.182.67:40429)
0810 18:06:09.833576 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:09.833620 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:09.834920 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:09.835468 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:09.835474 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:09.836065 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:09.836718 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:09.836734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.837730 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.837736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:09.838976 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:09.838989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.839514 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.839519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:09.839790 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:09.840504 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:09.840521 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:09.842420 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:09.844475 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:09.844482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:09.844498 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:09.844817 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:09.845103 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:09.845105 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:09.845106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:09.845297 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:09.845695 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:09.845701 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:09.846000 (+ 299us) client_negotiation.cc:769] Sending connection context
0810 18:06:09.846175 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 18:06:09.846515 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":117,"threads_started":1}
I20260810 18:06:09.849046 23422 heartbeater.cc:347] Connected to a master server at 127.21.182.126:33241
I20260810 18:06:09.849391 23422 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:09.850063 23422 heartbeater.cc:511] Master 127.21.182.126:33241 requested a full tablet report, sending...
I20260810 18:06:09.851436 22884 ts_manager.cc:194] Registered new tserver with Master: 5bfb8b68c2d84b609636263aae0773bf (127.21.182.67:35755)
I20260810 18:06:09.851995 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:06:09.852226 22884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:40429
I20260810 18:06:09.865514 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:09.856545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.21.182.126:33241)
0810 18:06:09.856725 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:09.856731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:09.856957 (+ 226us) server_negotiation.cc:407] Connection header received
0810 18:06:09.857146 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:09.857149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:09.857207 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:09.857313 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:09.858182 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.858800 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.859920 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.860179 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.860312 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:09.862029 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:09.862036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:09.862045 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:09.862076 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:09.863527 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:09.864109 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:09.864113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:09.864115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:09.864216 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:09.864518 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:09.864522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:09.864523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:09.864785 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:09.864937 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:09.865207 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 18:06:09.865336 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:06:09.873870 22884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:06:09.877269 22884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:09.894487 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:09.888210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:34823 (local address 127.0.0.1:35742)
0810 18:06:09.888979 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:09.889040 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:09.889190 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:09.890294 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:09.890301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:09.890327 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:09.890516 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:09.890523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.891634 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.891639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:09.892903 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:09.892913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.893840 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.893844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:09.893998 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:09.894042 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 18:06:09.894194 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 18:06:09.894338 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260810 18:06:09.894563 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:09.888345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35742 (local address 127.21.182.66:34823)
0810 18:06:09.889513 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:09.889528 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:09.889556 (+ 28us) server_negotiation.cc:407] Connection header received
0810 18:06:09.889639 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:09.889647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:09.889902 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:09.890061 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:09.890658 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.891500 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.893050 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.893698 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.893847 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:09.894071 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:09.894227 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 18:06:09.894328 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1057,"thread_start_us":188,"threads_started":1}
I20260810 18:06:09.896088 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:09.888321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:35755 (local address 127.0.0.1:60204)
0810 18:06:09.889637 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:09.889651 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:09.889731 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:09.890471 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:09.890474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:09.890489 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:09.890571 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:09.890581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.892425 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.892428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:09.894372 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:09.894385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.895618 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.895622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:09.895718 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:09.895746 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:06:09.895808 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 18:06:09.895936 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1256,"mutex_wait_us":119,"thread_start_us":96,"threads_started":1}
I20260810 18:06:09.896935 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:09.889194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.21.182.67:35755)
0810 18:06:09.889653 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:09.889694 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:09.889721 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:06:09.889970 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:09.889976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:09.890274 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:09.890476 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:09.890723 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.892131 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.894534 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.895460 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.896399 (+ 939us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:09.896549 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:09.896626 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 18:06:09.896734 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":143,"threads_started":1}
I20260810 18:06:09.897048 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:09.888636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:42367 (local address 127.0.0.1:56382)
0810 18:06:09.890112 (+ 1476us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:09.890127 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:09.890238 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:09.890693 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:09.890697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:09.890717 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:09.890802 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:09.890808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.892381 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.892388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:09.894372 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:09.894385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.896625 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.896629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:09.896717 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:09.896740 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:06:09.896838 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 18:06:09.896906 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1413,"mutex_wait_us":140,"thread_start_us":99,"threads_started":1}
I20260810 18:06:09.897483 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:09.889175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.21.182.65:42367)
0810 18:06:09.889979 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:09.889987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:09.890249 (+ 262us) server_negotiation.cc:407] Connection header received
0810 18:06:09.890355 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:09.890360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:09.890541 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:09.890707 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:09.890897 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.892141 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.894534 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:09.895819 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:09.895941 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:09.896577 (+ 636us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:09.896891 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 18:06:09.897336 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":137,"threads_started":1}
I20260810 18:06:09.898831 23183 tablet_service.cc:1467] Processing CreateTablet for tablet bd547847deaa42e4a8635d40b341bccb (DEFAULT_TABLE table=test-table [id=40754ea2be9543649e73bd76809c9480]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:09.900259 23183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd547847deaa42e4a8635d40b341bccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:09.900090 23344 tablet_service.cc:1467] Processing CreateTablet for tablet bd547847deaa42e4a8635d40b341bccb (DEFAULT_TABLE table=test-table [id=40754ea2be9543649e73bd76809c9480]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:09.901240 23344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd547847deaa42e4a8635d40b341bccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:09.901542 23025 tablet_service.cc:1467] Processing CreateTablet for tablet bd547847deaa42e4a8635d40b341bccb (DEFAULT_TABLE table=test-table [id=40754ea2be9543649e73bd76809c9480]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:09.902582 23025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd547847deaa42e4a8635d40b341bccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:09.911146 23441 tablet_bootstrap.cc:492] T bd547847deaa42e4a8635d40b341bccb P b4b1cc5320bd41c4850fe59c51c010f2: Bootstrap starting.
I20260810 18:06:09.911198 23442 tablet_bootstrap.cc:492] T bd547847deaa42e4a8635d40b341bccb P ea99622d19b7474687cec6ea8e714a89: Bootstrap starting.
I20260810 18:06:09.911222 23440 tablet_bootstrap.cc:492] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf: Bootstrap starting.
I20260810 18:06:09.913997 23442 tablet_bootstrap.cc:654] T bd547847deaa42e4a8635d40b341bccb P ea99622d19b7474687cec6ea8e714a89: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:09.914004 23441 tablet_bootstrap.cc:654] T bd547847deaa42e4a8635d40b341bccb P b4b1cc5320bd41c4850fe59c51c010f2: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:09.914043 23440 tablet_bootstrap.cc:654] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:09.914906 23440 log.cc:824] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:09.915124 23442 log.cc:824] T bd547847deaa42e4a8635d40b341bccb P ea99622d19b7474687cec6ea8e714a89: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:09.916656 23441 log.cc:824] T bd547847deaa42e4a8635d40b341bccb P b4b1cc5320bd41c4850fe59c51c010f2: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:09.917683 23440 tablet_bootstrap.cc:492] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf: No bootstrap required, opened a new log
I20260810 18:06:09.917744 23442 tablet_bootstrap.cc:492] T bd547847deaa42e4a8635d40b341bccb P ea99622d19b7474687cec6ea8e714a89: No bootstrap required, opened a new log
I20260810 18:06:09.917955 23442 ts_tablet_manager.cc:1397] T bd547847deaa42e4a8635d40b341bccb P ea99622d19b7474687cec6ea8e714a89: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:06:09.917956 23440 ts_tablet_manager.cc:1397] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260810 18:06:09.919339 23441 tablet_bootstrap.cc:492] T bd547847deaa42e4a8635d40b341bccb P b4b1cc5320bd41c4850fe59c51c010f2: No bootstrap required, opened a new log
I20260810 18:06:09.919600 23441 ts_tablet_manager.cc:1397] T bd547847deaa42e4a8635d40b341bccb P b4b1cc5320bd41c4850fe59c51c010f2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 18:06:09.921728 23440 raft_consensus.cc:348] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4b1cc5320bd41c4850fe59c51c010f2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34823 } } peers { permanent_uuid: "5bfb8b68c2d84b609636263aae0773bf" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 35755 } } peers { permanent_uuid: "ea99622d19b7474687cec6ea8e714a89" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42367 } }
I20260810 18:06:09.922029 23440 raft_consensus.cc:374] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:09.922093 23440 raft_consensus.cc:729] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bfb8b68c2d84b609636263aae0773bf, State: Initialized, Role: FOLLOWER
I20260810 18:06:09.922605 23440 consensus_queue.cc:260] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4b1cc5320bd41c4850fe59c51c010f2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34823 } } peers { permanent_uuid: "5bfb8b68c2d84b609636263aae0773bf" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 35755 } } peers { permanent_uuid: "ea99622d19b7474687cec6ea8e714a89" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42367 } }
I20260810 18:06:09.923383 23422 heartbeater.cc:503] Master 127.21.182.126:33241 was elected leader, sending a full tablet report...
I20260810 18:06:09.923875 23440 ts_tablet_manager.cc:1428] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:06:09.926079 23442 raft_consensus.cc:348] T bd547847deaa42e4a8635d40b341bccb P ea99622d19b7474687cec6ea8e714a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4b1cc5320bd41c4850fe59c51c010f2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34823 } } peers { permanent_uuid: "5bfb8b68c2d84b609636263aae0773bf" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 35755 } } peers { permanent_uuid: "ea99622d19b7474687cec6ea8e714a89" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42367 } }
I20260810 18:06:09.926412 23442 raft_consensus.cc:374] T bd547847deaa42e4a8635d40b341bccb P ea99622d19b7474687cec6ea8e714a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:09.926503 23442 raft_consensus.cc:729] T bd547847deaa42e4a8635d40b341bccb P ea99622d19b7474687cec6ea8e714a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea99622d19b7474687cec6ea8e714a89, State: Initialized, Role: FOLLOWER
I20260810 18:06:09.926438 23441 raft_consensus.cc:348] T bd547847deaa42e4a8635d40b341bccb P b4b1cc5320bd41c4850fe59c51c010f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4b1cc5320bd41c4850fe59c51c010f2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34823 } } peers { permanent_uuid: "5bfb8b68c2d84b609636263aae0773bf" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 35755 } } peers { permanent_uuid: "ea99622d19b7474687cec6ea8e714a89" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42367 } }
I20260810 18:06:09.926823 23441 raft_consensus.cc:374] T bd547847deaa42e4a8635d40b341bccb P b4b1cc5320bd41c4850fe59c51c010f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:09.926887 23441 raft_consensus.cc:729] T bd547847deaa42e4a8635d40b341bccb P b4b1cc5320bd41c4850fe59c51c010f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4b1cc5320bd41c4850fe59c51c010f2, State: Initialized, Role: FOLLOWER
I20260810 18:06:09.927198 23442 consensus_queue.cc:260] T bd547847deaa42e4a8635d40b341bccb P ea99622d19b7474687cec6ea8e714a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4b1cc5320bd41c4850fe59c51c010f2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34823 } } peers { permanent_uuid: "5bfb8b68c2d84b609636263aae0773bf" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 35755 } } peers { permanent_uuid: "ea99622d19b7474687cec6ea8e714a89" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42367 } }
I20260810 18:06:09.927291 23441 consensus_queue.cc:260] T bd547847deaa42e4a8635d40b341bccb P b4b1cc5320bd41c4850fe59c51c010f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4b1cc5320bd41c4850fe59c51c010f2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34823 } } peers { permanent_uuid: "5bfb8b68c2d84b609636263aae0773bf" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 35755 } } peers { permanent_uuid: "ea99622d19b7474687cec6ea8e714a89" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42367 } }
I20260810 18:06:09.928701 23263 heartbeater.cc:503] Master 127.21.182.126:33241 was elected leader, sending a full tablet report...
I20260810 18:06:09.928704 23442 ts_tablet_manager.cc:1428] T bd547847deaa42e4a8635d40b341bccb P ea99622d19b7474687cec6ea8e714a89: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260810 18:06:09.928805 23441 ts_tablet_manager.cc:1428] T bd547847deaa42e4a8635d40b341bccb P b4b1cc5320bd41c4850fe59c51c010f2: Time spent starting tablet: real 0.009s user 0.002s sys 0.008s
I20260810 18:06:09.929109 23104 heartbeater.cc:503] Master 127.21.182.126:33241 was elected leader, sending a full tablet report...
W20260810 18:06:10.068413 23265 tablet.cc:2369] T bd547847deaa42e4a8635d40b341bccb P b4b1cc5320bd41c4850fe59c51c010f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:06:10.081624 23423 tablet.cc:2369] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:06:10.088557 23105 tablet.cc:2369] T bd547847deaa42e4a8635d40b341bccb P ea99622d19b7474687cec6ea8e714a89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:06:10.194903 23446 raft_consensus.cc:482] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:10.195309 23446 raft_consensus.cc:504] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4b1cc5320bd41c4850fe59c51c010f2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34823 } } peers { permanent_uuid: "5bfb8b68c2d84b609636263aae0773bf" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 35755 } } peers { permanent_uuid: "ea99622d19b7474687cec6ea8e714a89" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42367 } }
I20260810 18:06:10.197060 23446 leader_election.cc:288] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4b1cc5320bd41c4850fe59c51c010f2 (127.21.182.66:34823), ea99622d19b7474687cec6ea8e714a89 (127.21.182.65:42367)
I20260810 18:06:10.201207 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:10.197326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:34823 (local address 127.21.182.67:56635)
0810 18:06:10.197632 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:10.197650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:10.197859 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:10.198171 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:10.198173 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:10.198191 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:10.198397 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:10.198404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:10.199173 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:10.199178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:10.200026 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:10.200036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:10.200697 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:10.200701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:10.200873 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:10.200893 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:06:10.200991 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 18:06:10.201070 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260810 18:06:10.201208 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:10.197426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:56635 (local address 127.21.182.66:34823)
0810 18:06:10.197560 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:10.197565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:10.197851 (+ 286us) server_negotiation.cc:407] Connection header received
0810 18:06:10.198004 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:10.198007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:10.198055 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:10.198149 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:10.198528 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:10.199046 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:10.200162 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:10.200567 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:10.200702 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:10.200759 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:10.201010 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 18:06:10.201084 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:06:10.201946 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:10.197588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:57123 (local address 127.21.182.65:42367)
0810 18:06:10.197758 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:10.197764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:10.198051 (+ 287us) server_negotiation.cc:407] Connection header received
0810 18:06:10.198230 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:10.198232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:10.198276 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:10.198367 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:10.198790 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:10.199287 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:10.200781 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:10.201210 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:10.201344 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:10.201436 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:10.201762 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 18:06:10.201839 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 18:06:10.201979 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:10.197367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:42367 (local address 127.21.182.67:57123)
0810 18:06:10.197921 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:10.197932 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:10.198090 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:10.198380 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:10.198383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:10.198400 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:10.198632 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:10.198653 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:10.199405 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:10.199410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:10.200634 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:10.200644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:10.201437 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:10.201443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:10.201601 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:10.201635 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 18:06:10.201741 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 18:06:10.201808 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":12,"thread_start_us":187,"threads_started":1}
I20260810 18:06:10.202204 23206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd547847deaa42e4a8635d40b341bccb" candidate_uuid: "5bfb8b68c2d84b609636263aae0773bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4b1cc5320bd41c4850fe59c51c010f2" is_pre_election: true
I20260810 18:06:10.202752 23206 raft_consensus.cc:2454] T bd547847deaa42e4a8635d40b341bccb P b4b1cc5320bd41c4850fe59c51c010f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bfb8b68c2d84b609636263aae0773bf in term 0.
I20260810 18:06:10.202888 23052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd547847deaa42e4a8635d40b341bccb" candidate_uuid: "5bfb8b68c2d84b609636263aae0773bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99622d19b7474687cec6ea8e714a89" is_pre_election: true
I20260810 18:06:10.203191 23052 raft_consensus.cc:2454] T bd547847deaa42e4a8635d40b341bccb P ea99622d19b7474687cec6ea8e714a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bfb8b68c2d84b609636263aae0773bf in term 0.
I20260810 18:06:10.203435 23289 leader_election.cc:302] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bfb8b68c2d84b609636263aae0773bf, b4b1cc5320bd41c4850fe59c51c010f2; no voters:
I20260810 18:06:10.203787 23446 raft_consensus.cc:2781] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:10.203876 23446 raft_consensus.cc:482] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:10.203967 23446 raft_consensus.cc:3037] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:10.205602 23446 raft_consensus.cc:504] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4b1cc5320bd41c4850fe59c51c010f2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34823 } } peers { permanent_uuid: "5bfb8b68c2d84b609636263aae0773bf" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 35755 } } peers { permanent_uuid: "ea99622d19b7474687cec6ea8e714a89" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42367 } }
I20260810 18:06:10.206000 23446 leader_election.cc:288] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [CANDIDATE]: Term 1 election: Requested vote from peers b4b1cc5320bd41c4850fe59c51c010f2 (127.21.182.66:34823), ea99622d19b7474687cec6ea8e714a89 (127.21.182.65:42367)
I20260810 18:06:10.206560 23206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd547847deaa42e4a8635d40b341bccb" candidate_uuid: "5bfb8b68c2d84b609636263aae0773bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4b1cc5320bd41c4850fe59c51c010f2"
I20260810 18:06:10.206712 23206 raft_consensus.cc:3037] T bd547847deaa42e4a8635d40b341bccb P b4b1cc5320bd41c4850fe59c51c010f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:10.206601 23052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd547847deaa42e4a8635d40b341bccb" candidate_uuid: "5bfb8b68c2d84b609636263aae0773bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea99622d19b7474687cec6ea8e714a89"
I20260810 18:06:10.206871 23052 raft_consensus.cc:3037] T bd547847deaa42e4a8635d40b341bccb P ea99622d19b7474687cec6ea8e714a89 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:10.208336 23206 raft_consensus.cc:2454] T bd547847deaa42e4a8635d40b341bccb P b4b1cc5320bd41c4850fe59c51c010f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bfb8b68c2d84b609636263aae0773bf in term 1.
I20260810 18:06:10.208441 23052 raft_consensus.cc:2454] T bd547847deaa42e4a8635d40b341bccb P ea99622d19b7474687cec6ea8e714a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bfb8b68c2d84b609636263aae0773bf in term 1.
I20260810 18:06:10.208734 23289 leader_election.cc:302] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bfb8b68c2d84b609636263aae0773bf, b4b1cc5320bd41c4850fe59c51c010f2; no voters:
I20260810 18:06:10.208958 23446 raft_consensus.cc:2781] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:10.209314 23446 raft_consensus.cc:686] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [term 1 LEADER]: Becoming Leader. State: Replica: 5bfb8b68c2d84b609636263aae0773bf, State: Running, Role: LEADER
I20260810 18:06:10.209677 23446 consensus_queue.cc:237] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4b1cc5320bd41c4850fe59c51c010f2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34823 } } peers { permanent_uuid: "5bfb8b68c2d84b609636263aae0773bf" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 35755 } } peers { permanent_uuid: "ea99622d19b7474687cec6ea8e714a89" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42367 } }
I20260810 18:06:10.213320 22883 catalog_manager.cc:5495] T bd547847deaa42e4a8635d40b341bccb P 5bfb8b68c2d84b609636263aae0773bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bfb8b68c2d84b609636263aae0773bf (127.21.182.67). New cstate: current_term: 1 leader_uuid: "5bfb8b68c2d84b609636263aae0773bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4b1cc5320bd41c4850fe59c51c010f2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bfb8b68c2d84b609636263aae0773bf" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 35755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea99622d19b7474687cec6ea8e714a89" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42367 } health_report { overall_health: UNKNOWN } } }
Aug 10 18:06:10 dist-test-slave-27sr krb5kdc[22835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.65@KRBTEST.COM
I20260810 18:06:10.275018 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:10.255199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.21.182.65:42367)
0810 18:06:10.255422 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:10.255426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:10.255531 (+ 105us) server_negotiation.cc:407] Connection header received
0810 18:06:10.255653 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:10.255655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:10.255701 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:10.255776 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:10.256564 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:10.257095 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:10.258323 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:10.258572 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:10.258682 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:10.270466 (+ 11784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:10.270485 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:10.270502 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:10.270543 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:10.272830 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:10.273425 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:10.273432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:10.273433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:10.273529 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:10.273900 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:10.273903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:10.273904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:10.274208 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:10.274469 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:10.274695 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 18:06:10.274840 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
Aug 10 18:06:10 dist-test-slave-27sr krb5kdc[22835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385170, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:06:10 dist-test-slave-27sr krb5kdc[22835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385170, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.182.65@KRBTEST.COM
I20260810 18:06:10.307102 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:10.297938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.21.182.65:42367)
0810 18:06:10.298140 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:10.298145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:10.298162 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:06:10.298398 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:10.298401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:10.298454 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:10.298537 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:10.299278 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:10.299880 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:10.300803 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:10.301039 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:10.301127 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:10.303719 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:10.303723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:10.303726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:10.303757 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:10.305175 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:10.305611 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:10.305615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:10.305616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:10.305699 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:10.306011 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:10.306015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:10.306016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:10.306230 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:10.306364 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:10.306725 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 18:06:10.306931 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
Aug 10 18:06:10 dist-test-slave-27sr krb5kdc[22835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:06:10 dist-test-slave-27sr krb5kdc[22835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.65@KRBTEST.COM
I20260810 18:06:10.339087 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:10.329460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.21.182.65:42367)
0810 18:06:10.329651 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:10.329655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:10.329676 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:06:10.329845 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:10.329848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:10.329896 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:10.329971 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:10.330747 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:10.331246 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:10.332209 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:10.332592 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:10.332681 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:10.335601 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:10.335604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:10.335607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:10.335635 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:10.337301 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:10.337796 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:10.337800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:10.337801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:10.337881 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:10.338177 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:10.338180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:10.338182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:10.338378 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:10.338519 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:10.338787 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 18:06:10.338914 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260810 18:06:10.340701 23003 tablet_service.cc:2908] Rejecting scan request for tablet bd547847deaa42e4a8635d40b341bccb: Uninitialized: clean time has not yet been initialized
I20260810 18:06:10.343444 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 22956
I20260810 18:06:10.354789 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 23109
I20260810 18:06:10.365517 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 23269
I20260810 18:06:10.374766 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 22850
2026-08-10T18:06:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2907 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:06:10 dist-test-slave-27sr krb5kdc[23464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:06:10 dist-test-slave-27sr krb5kdc[23464](info): set up 2 sockets
Aug 10 18:06:10 dist-test-slave-27sr krb5kdc[23464](info): commencing operation
krb5kdc: starting...
W20260810 18:06:11.587146 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:06:11 dist-test-slave-27sr krb5kdc[23464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:06:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:06:11.726455 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:44695
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:44695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:11.844259 23478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:11.844588 23478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:11.844667 23478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:11.848688 23478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:11.848788 23478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 18:06:11.848819 23478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:11.848865 23478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:11.848902 23478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:11.848938 23478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:11.854350 23478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:44695
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:44695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:11.855526 23478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:11.856635 23478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:11.865777 23487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:11.865792 23485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:11.865752 23483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:11.866067 23478 server_base.cc:1034] running on GCE node
I20260810 18:06:11.866816 23478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:11.868088 23478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:11.869269 23478 hybrid_clock.cc:648] HybridClock initialized: now 1786385171869247 us; error 33 us; skew 500 ppm
Aug 10 18:06:11 dist-test-slave-27sr krb5kdc[23464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385171, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:11.872203 23478 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:06:11.873114 23478 webserver.cc:460] Webserver started at http://127.21.182.126:38647/ using document root <none> and password file <none>
I20260810 18:06:11.873685 23478 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:11.873747 23478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:11.873916 23478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:11.876005 23478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "82d45692ca63498db7775be835977f35"
format_stamp: "Formatted at 2026-08-10 18:06:11 on dist-test-slave-27sr"
server_key: "6667e0535ed742318f47eb82912165a8"
server_key_iv: "aa5c62ffcb71a6fbf23fea4e375547e8"
server_key_version: "encryptionkey@0"
I20260810 18:06:11.876550 23478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "82d45692ca63498db7775be835977f35"
format_stamp: "Formatted at 2026-08-10 18:06:11 on dist-test-slave-27sr"
server_key: "6667e0535ed742318f47eb82912165a8"
server_key_iv: "aa5c62ffcb71a6fbf23fea4e375547e8"
server_key_version: "encryptionkey@0"
I20260810 18:06:11.880362 23478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:06:11.882824 23494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:11.884310 23478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:11.884459 23478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "82d45692ca63498db7775be835977f35"
format_stamp: "Formatted at 2026-08-10 18:06:11 on dist-test-slave-27sr"
server_key: "6667e0535ed742318f47eb82912165a8"
server_key_iv: "aa5c62ffcb71a6fbf23fea4e375547e8"
server_key_version: "encryptionkey@0"
I20260810 18:06:11.884589 23478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:11.909185 23478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:11.912813 23478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:11.913045 23478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:11.922294 23478 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:44695
I20260810 18:06:11.922313 23565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:44695 every 8 connection(s)
I20260810 18:06:11.923436 23478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:11.926595 23566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:11.933306 23566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35: Bootstrap starting.
I20260810 18:06:11.933738 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 23478
I20260810 18:06:11.933843 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:11.934139 22233 external_mini_cluster.cc:1428] Setting key 4c4dca7974fd681ba56dc1a8bb0b4f82
I20260810 18:06:11.936273 23566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:11.937048 23566 log.cc:824] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:11.939435 23566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35: No bootstrap required, opened a new log
Aug 10 18:06:11 dist-test-slave-27sr krb5kdc[23464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:11.944259 23566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d45692ca63498db7775be835977f35" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44695 } }
I20260810 18:06:11.944573 23566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:11.944634 23566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d45692ca63498db7775be835977f35, State: Initialized, Role: FOLLOWER
I20260810 18:06:11.945166 23566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d45692ca63498db7775be835977f35" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44695 } }
I20260810 18:06:11.945351 23566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:11.945422 23566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:11.945535 23566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:11.946925 23566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d45692ca63498db7775be835977f35" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44695 } }
I20260810 18:06:11.947328 23566 leader_election.cc:302] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82d45692ca63498db7775be835977f35; no voters:
I20260810 18:06:11.947714 23566 leader_election.cc:288] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:11.947881 23572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:11.948210 23572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [term 1 LEADER]: Becoming Leader. State: Replica: 82d45692ca63498db7775be835977f35, State: Running, Role: LEADER
I20260810 18:06:11.948563 23572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d45692ca63498db7775be835977f35" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44695 } }
I20260810 18:06:11.949257 23566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:11.950445 23574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82d45692ca63498db7775be835977f35. Latest consensus state: current_term: 1 leader_uuid: "82d45692ca63498db7775be835977f35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d45692ca63498db7775be835977f35" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44695 } } }
I20260810 18:06:11.950568 23574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:11.950898 23573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82d45692ca63498db7775be835977f35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d45692ca63498db7775be835977f35" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44695 } } }
I20260810 18:06:11.951009 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:11.935674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.21.182.126:44695)
0810 18:06:11.936219 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:11.936237 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:11.936276 (+ 39us) server_negotiation.cc:407] Connection header received
0810 18:06:11.937053 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:11.937073 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:11.937572 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:11.937994 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:11.938986 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:11.940135 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:11.941374 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:11.941849 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:11.941966 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:11.944930 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:11.944936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:11.944954 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:11.944987 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:11.947285 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:11.947850 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:11.947856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:11.947862 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:11.948089 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:11.948507 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:11.948510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:11.948512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:11.949029 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:11.949223 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:11.949564 (+ 341us) server_negotiation.cc:299] Negotiation successful
0810 18:06:11.949887 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":185,"threads_started":1}
I20260810 18:06:11.951025 23573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35 [sys.catalog]: This master's current role is: LEADER
W20260810 18:06:11.954795 23590 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:06:11.954892 23590 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:06:11.954990 23584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:11.955981 23584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:11.962584 23584 catalog_manager.cc:1347] Generated new cluster ID: b05f8e441bb4433cbe0de53e91f10211
I20260810 18:06:11.962678 23584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:11.979624 23584 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:11.980785 23584 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:11.994058 23584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35: Generated new TSK 0
I20260810 18:06:11.994843 23584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:06:12.035830 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:44695
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:12.164019 23596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:12.164388 23596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:12.164480 23596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:12.168869 23596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:12.168982 23596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:12.169060 23596 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:06:12.169150 23596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:12.175063 23596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.182.126:44695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:12.176455 23596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:12.177625 23596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:12.188211 23605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:12.188222 23602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:12.188203 23601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:12.188585 23596 server_base.cc:1034] running on GCE node
I20260810 18:06:12.189100 23596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:12.189865 23596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:12.191063 23596 hybrid_clock.cc:648] HybridClock initialized: now 1786385172191043 us; error 37 us; skew 500 ppm
Aug 10 18:06:12 dist-test-slave-27sr krb5kdc[23464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385172, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:12.194249 23596 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:06:12.195269 23596 webserver.cc:460] Webserver started at http://127.21.182.65:44069/ using document root <none> and password file <none>
I20260810 18:06:12.195972 23596 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:12.196071 23596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:12.196327 23596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:12.198535 23596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "64fea87ffe1447a892bd572de1957026"
format_stamp: "Formatted at 2026-08-10 18:06:12 on dist-test-slave-27sr"
server_key: "b33b113b315c9abe6e34cbd03abffad8"
server_key_iv: "99f09dcf5315059aea2675a10149a571"
server_key_version: "encryptionkey@0"
I20260810 18:06:12.199230 23596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "64fea87ffe1447a892bd572de1957026"
format_stamp: "Formatted at 2026-08-10 18:06:12 on dist-test-slave-27sr"
server_key: "b33b113b315c9abe6e34cbd03abffad8"
server_key_iv: "99f09dcf5315059aea2675a10149a571"
server_key_version: "encryptionkey@0"
I20260810 18:06:12.203473 23596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 18:06:12.206347 23612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:12.207721 23596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:12.207945 23596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "64fea87ffe1447a892bd572de1957026"
format_stamp: "Formatted at 2026-08-10 18:06:12 on dist-test-slave-27sr"
server_key: "b33b113b315c9abe6e34cbd03abffad8"
server_key_iv: "99f09dcf5315059aea2675a10149a571"
server_key_version: "encryptionkey@0"
I20260810 18:06:12.208083 23596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:12.225669 23596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:12.229036 23596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:12.229265 23596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:12.230937 23596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:12.231016 23596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:12.231088 23596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:12.231132 23596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:06:12 dist-test-slave-27sr krb5kdc[23464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385172, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:12.243142 23596 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:44373
I20260810 18:06:12.243254 23738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:44373 every 8 connection(s)
I20260810 18:06:12.244257 23596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:12.244529 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 23596
I20260810 18:06:12.244679 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:06:12.245010 22233 external_mini_cluster.cc:1428] Setting key 99113b111b76b094441ee1fa1095d0f2
I20260810 18:06:12.247874 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.231853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:52867 (local address 127.21.182.126:44695)
0810 18:06:12.232039 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:12.232043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:12.233097 (+ 1054us) server_negotiation.cc:407] Connection header received
0810 18:06:12.234107 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:12.234111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:12.234171 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:12.234257 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:12.235700 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.236334 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.238362 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.238793 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.238932 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.243594 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:12.243599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:12.243601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:12.243641 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:12.245499 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:12.246187 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:12.246191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:12.246192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:12.246304 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:12.246690 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:12.246694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:12.246696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:12.246902 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:12.247019 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:12.247548 (+ 529us) server_negotiation.cc:299] Negotiation successful
0810 18:06:12.247702 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 18:06:12.248900 23621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.232391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44695 (local address 127.21.182.65:52867)
0810 18:06:12.232934 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:12.232980 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:12.233887 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:12.234586 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:12.234595 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:12.235000 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:12.235501 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.235515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.236489 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.236494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.238185 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.238203 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.239020 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.239027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.239297 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.241242 (+ 1945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:12.241269 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:12.243371 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:12.245662 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:12.245669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:12.245689 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:12.246080 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:12.246419 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:12.246423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:12.246425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:12.246580 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:12.247017 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:12.247024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:12.247291 (+ 267us) client_negotiation.cc:769] Sending connection context
0810 18:06:12.247503 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 18:06:12.247867 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":150,"threads_started":1}
I20260810 18:06:12.250511 23739 heartbeater.cc:347] Connected to a master server at 127.21.182.126:44695
I20260810 18:06:12.250705 23739 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:12.251315 23739 heartbeater.cc:511] Master 127.21.182.126:44695 requested a full tablet report, sending...
I20260810 18:06:12.253075 23512 ts_manager.cc:194] Registered new tserver with Master: 64fea87ffe1447a892bd572de1957026 (127.21.182.65:44373)
I20260810 18:06:12.254408 23512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:52867
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
I20260810 18:06:12.263345 23748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.253399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44695 (local address 127.21.182.65:34237)
0810 18:06:12.255162 (+ 1763us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:12.255175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:12.255301 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:12.255576 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:12.255579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:12.255826 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:12.255987 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.255996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.257241 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.257243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.257806 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.257814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.258295 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.258302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.258424 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.259043 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:12.259055 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:12.259555 (+ 500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:12.261512 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:12.261518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:12.261523 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:12.261813 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:12.262104 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:12.262110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:12.262115 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:12.262280 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:12.262710 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:12.262716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:12.262853 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 18:06:12.262983 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 18:06:12.263183 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1673,"thread_start_us":101,"threads_started":1}
I20260810 18:06:12.263435 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.253484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:34237 (local address 127.21.182.126:44695)
0810 18:06:12.253643 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:12.253650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:12.255258 (+ 1608us) server_negotiation.cc:407] Connection header received
0810 18:06:12.255417 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:12.255420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:12.255472 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:12.255558 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:12.256129 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.257012 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.257918 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.258133 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.258274 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.259682 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:12.259685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:12.259687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:12.259706 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:12.261388 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:12.261923 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:12.261926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:12.261927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:12.262007 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:12.262393 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:12.262396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:12.262398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:12.262612 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:12.262692 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:12.263067 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 18:06:12.263227 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:06:12.279662 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:44695
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:12.404250 23750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:12.404554 23750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:12.404603 23750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:12.408725 23750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:12.408792 23750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:12.408833 23750 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:06:12.408866 23750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:12.414731 23750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.182.126:44695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:12.415882 23750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:12.417026 23750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:12.428164 23756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:12.428309 23750 server_base.cc:1034] running on GCE node
W20260810 18:06:12.428154 23755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:12.428181 23758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:12.428948 23750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:12.429692 23750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:12.430896 23750 hybrid_clock.cc:648] HybridClock initialized: now 1786385172430880 us; error 34 us; skew 500 ppm
Aug 10 18:06:12 dist-test-slave-27sr krb5kdc[23464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385172, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:12.434084 23750 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:06:12.435163 23750 webserver.cc:460] Webserver started at http://127.21.182.66:42487/ using document root <none> and password file <none>
I20260810 18:06:12.435803 23750 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:12.435905 23750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:12.436193 23750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:12.438930 23750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "e0cace1b79ea497d9842d4caab8b9bea"
format_stamp: "Formatted at 2026-08-10 18:06:12 on dist-test-slave-27sr"
server_key: "3c93427545d7f6a7853b709db80fd8fb"
server_key_iv: "56edf87fbf7d9bf8c6ddd41e336babf1"
server_key_version: "encryptionkey@0"
I20260810 18:06:12.439566 23750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "e0cace1b79ea497d9842d4caab8b9bea"
format_stamp: "Formatted at 2026-08-10 18:06:12 on dist-test-slave-27sr"
server_key: "3c93427545d7f6a7853b709db80fd8fb"
server_key_iv: "56edf87fbf7d9bf8c6ddd41e336babf1"
server_key_version: "encryptionkey@0"
I20260810 18:06:12.444108 23750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260810 18:06:12.447026 23765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:12.448576 23750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:12.448750 23750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "e0cace1b79ea497d9842d4caab8b9bea"
format_stamp: "Formatted at 2026-08-10 18:06:12 on dist-test-slave-27sr"
server_key: "3c93427545d7f6a7853b709db80fd8fb"
server_key_iv: "56edf87fbf7d9bf8c6ddd41e336babf1"
server_key_version: "encryptionkey@0"
I20260810 18:06:12.448899 23750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:12.462625 23750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:12.466544 23750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:12.466737 23750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:12.468572 23750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:12.468638 23750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:12.468684 23750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:12.468741 23750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:06:12 dist-test-slave-27sr krb5kdc[23464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385172, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:12.480630 23750 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:42603
I20260810 18:06:12.480721 23900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:42603 every 8 connection(s)
I20260810 18:06:12.481643 23750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:06:12.483383 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.469450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:45811 (local address 127.21.182.126:44695)
0810 18:06:12.469620 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:12.469626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:12.470635 (+ 1009us) server_negotiation.cc:407] Connection header received
0810 18:06:12.471779 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:12.471782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:12.471846 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:12.471965 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:12.473537 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.474194 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.475429 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.475733 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.475863 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.479203 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:12.479207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:12.479210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:12.479255 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:12.481087 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:12.481711 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:12.481715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:12.481717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:12.481823 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:12.482250 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:12.482254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:12.482256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:12.482518 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:12.482627 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:12.483128 (+ 501us) server_negotiation.cc:299] Negotiation successful
0810 18:06:12.483250 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:06:12.484328 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.469942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44695 (local address 127.21.182.66:45811)
0810 18:06:12.470440 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:12.470496 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:12.471462 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:12.472155 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:12.472162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:12.472704 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:12.473374 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.473400 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.474347 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.474351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.475275 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.475286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.475949 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.475954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.476195 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.477003 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:12.477030 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:12.478985 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:12.481225 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:12.481232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:12.481249 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:12.481580 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:12.481934 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:12.481938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:12.481940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:12.482121 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:12.482636 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:12.482642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:12.482890 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 18:06:12.483067 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 18:06:12.483346 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":143,"threads_started":1}
I20260810 18:06:12.485671 23901 heartbeater.cc:347] Connected to a master server at 127.21.182.126:44695
I20260810 18:06:12.485833 23901 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:12.486512 23901 heartbeater.cc:511] Master 127.21.182.126:44695 requested a full tablet report, sending...
I20260810 18:06:12.487922 23513 ts_manager.cc:194] Registered new tserver with Master: e0cace1b79ea497d9842d4caab8b9bea (127.21.182.66:42603)
I20260810 18:06:12.488268 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 23750
I20260810 18:06:12.488415 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:06:12.488782 22233 external_mini_cluster.cc:1428] Setting key 16b9685f6ffddc8daf115ab79225f2d1
I20260810 18:06:12.488891 23513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:45811
I20260810 18:06:12.498409 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.488639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44695 (local address 127.21.182.66:54567)
0810 18:06:12.489053 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:12.489071 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:12.489173 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:12.489881 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:12.489885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:12.490121 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:12.490296 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.490306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.491346 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.491353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.492368 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.492383 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.493191 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.493195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.493338 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.493948 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:12.493960 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:12.494338 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:12.496521 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:12.496524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:12.496526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:12.496736 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:12.497120 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:12.497123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:12.497125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:12.497223 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:12.497865 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:12.497869 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:12.497962 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 18:06:12.498085 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 18:06:12.498244 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":142,"threads_started":1}
I20260810 18:06:12.498538 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.489420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:54567 (local address 127.21.182.126:44695)
0810 18:06:12.489566 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:12.489571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:12.489588 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:06:12.489630 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:12.489633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:12.489688 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:12.489868 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:12.490445 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.491184 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.492538 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.493036 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.493159 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.494482 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:12.494489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:12.494495 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:12.494530 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:12.496397 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:12.496847 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:12.496853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:12.496858 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:12.496970 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:12.497337 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:12.497343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:12.497346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:12.497723 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:12.497858 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:12.498180 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 18:06:12.498325 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:06:12.523419 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:44695
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:12.647784 23913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:12.648196 23913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:12.648254 23913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:12.652704 23913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:12.652779 23913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:12.652823 23913 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:06:12.652858 23913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:12.659106 23913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.182.126:44695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:12.660290 23913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:12.661375 23913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:12.672740 23921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:12.672740 23920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:12.672901 23923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:12.673242 23913 server_base.cc:1034] running on GCE node
I20260810 18:06:12.673754 23913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:12.674536 23913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:12.675750 23913 hybrid_clock.cc:648] HybridClock initialized: now 1786385172675709 us; error 54 us; skew 500 ppm
Aug 10 18:06:12 dist-test-slave-27sr krb5kdc[23464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385172, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:12.679020 23913 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:06:12.680131 23913 webserver.cc:460] Webserver started at http://127.21.182.67:44699/ using document root <none> and password file <none>
I20260810 18:06:12.680778 23913 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:12.680897 23913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:12.681200 23913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:12.683627 23913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "75cf81b973224f4182b13ef439edae8d"
format_stamp: "Formatted at 2026-08-10 18:06:12 on dist-test-slave-27sr"
server_key: "b82dd9d5e0025484337a56dd500b3e14"
server_key_iv: "f0f83ec5a8dc548b2326bc99ded588a1"
server_key_version: "encryptionkey@0"
I20260810 18:06:12.684391 23913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "75cf81b973224f4182b13ef439edae8d"
format_stamp: "Formatted at 2026-08-10 18:06:12 on dist-test-slave-27sr"
server_key: "b82dd9d5e0025484337a56dd500b3e14"
server_key_iv: "f0f83ec5a8dc548b2326bc99ded588a1"
server_key_version: "encryptionkey@0"
I20260810 18:06:12.689190 23913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:06:12.692106 23932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:12.693491 23913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:12.693646 23913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "75cf81b973224f4182b13ef439edae8d"
format_stamp: "Formatted at 2026-08-10 18:06:12 on dist-test-slave-27sr"
server_key: "b82dd9d5e0025484337a56dd500b3e14"
server_key_iv: "f0f83ec5a8dc548b2326bc99ded588a1"
server_key_version: "encryptionkey@0"
I20260810 18:06:12.693778 23913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:12.707868 23913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:12.712193 23913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:12.712422 23913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:12.714089 23913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:12.714146 23913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:12.714213 23913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:12.714246 23913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:06:12 dist-test-slave-27sr krb5kdc[23464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385172, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:12.727345 23913 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:36273
I20260810 18:06:12.728080 24066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:36273 every 8 connection(s)
I20260810 18:06:12.728703 23913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:06:12.730536 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.715150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:43473 (local address 127.21.182.126:44695)
0810 18:06:12.715321 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:12.715325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:12.716505 (+ 1180us) server_negotiation.cc:407] Connection header received
0810 18:06:12.717917 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:12.717922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:12.717998 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:12.718107 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:12.719983 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.720919 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.722383 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.722849 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.723000 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.726157 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:12.726161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:12.726164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:12.726198 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:12.728056 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:12.728738 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:12.728742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:12.728761 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:12.728874 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:12.729329 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:12.729333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:12.729335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:12.729589 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:12.729691 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:12.730290 (+ 599us) server_negotiation.cc:299] Negotiation successful
0810 18:06:12.730411 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:06:12.731405 23943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.715773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44695 (local address 127.21.182.67:43473)
0810 18:06:12.716329 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:12.716375 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:12.717608 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:12.718358 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:12.718369 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:12.718932 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:12.719767 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.719790 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.721076 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.721080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.722231 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.722241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.723069 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.723073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.723244 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.723978 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:12.724004 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:12.725955 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:12.728206 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:12.728215 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:12.728233 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:12.728597 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:12.728971 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:12.728974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:12.728976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:12.729201 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:12.729752 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:12.729759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:12.730031 (+ 272us) client_negotiation.cc:769] Sending connection context
0810 18:06:12.730194 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 18:06:12.730472 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":134,"threads_started":1}
I20260810 18:06:12.732242 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 23913
I20260810 18:06:12.732388 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:06:12.732678 22233 external_mini_cluster.cc:1428] Setting key 9207f3ffca287eae19507cf77a21143e
I20260810 18:06:12.732843 24067 heartbeater.cc:347] Connected to a master server at 127.21.182.126:44695
I20260810 18:06:12.732990 24067 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:12.733705 24067 heartbeater.cc:511] Master 127.21.182.126:44695 requested a full tablet report, sending...
I20260810 18:06:12.734920 23513 ts_manager.cc:194] Registered new tserver with Master: 75cf81b973224f4182b13ef439edae8d (127.21.182.67:36273)
I20260810 18:06:12.735654 23513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:43473
I20260810 18:06:12.743029 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.735630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:39047 (local address 127.21.182.126:44695)
0810 18:06:12.735777 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:12.735782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:12.736058 (+ 276us) server_negotiation.cc:407] Connection header received
0810 18:06:12.736179 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:12.736182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:12.736225 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:12.736303 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:12.736744 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.737481 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.738345 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.738561 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.738656 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.739939 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:12.739943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:12.739946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:12.739970 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:12.741367 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:12.741834 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:12.741838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:12.741839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:12.741942 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:12.742191 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:12.742193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:12.742194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:12.742384 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:12.742471 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:12.742773 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 18:06:12.742901 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:06:12.743242 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.735505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44695 (local address 127.21.182.67:39047)
0810 18:06:12.735977 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:12.735994 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:12.736099 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:12.736316 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:12.736318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:12.736512 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:12.736613 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.736618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.737614 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.737616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.738228 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.738238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.738676 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.738679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.738778 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.739366 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:12.739381 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:12.739764 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:12.741477 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:12.741480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:12.741482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:12.741689 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:12.742024 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:12.742026 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:12.742028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:12.742112 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:12.742515 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:12.742518 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:12.742610 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 18:06:12.742708 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 18:06:12.742843 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":163,"threads_started":1}
I20260810 18:06:12.748497 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:06:12 dist-test-slave-27sr krb5kdc[23464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:06:12 dist-test-slave-27sr krb5kdc[23464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:12.779414 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.770242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.21.182.126:44695)
0810 18:06:12.770390 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:12.770394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:12.770406 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:06:12.770453 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:12.770455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:12.770496 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:12.770564 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:12.771440 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.772084 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.772953 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.773171 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.773294 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.776215 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:12.776239 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:12.776241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:12.776273 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:12.777657 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:12.778143 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:12.778146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:12.778147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:12.778252 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:12.778516 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:12.778518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:12.778520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:12.778726 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:12.778845 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:12.779082 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 18:06:12.779186 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:06:12.782877 23513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:06:12.786231 23513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:12.803450 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.796506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.21.182.66:42603)
0810 18:06:12.797996 (+ 1490us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:12.798010 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:12.798037 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:06:12.798134 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:12.798141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:12.798384 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:12.799149 (+ 765us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:12.799328 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.800557 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.802032 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.802803 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.802946 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.803115 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:12.803178 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 18:06:12.803294 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1361,"thread_start_us":129,"threads_started":1}
I20260810 18:06:12.803928 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.797237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.21.182.65:44373)
0810 18:06:12.797738 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:12.797747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:12.798170 (+ 423us) server_negotiation.cc:407] Connection header received
0810 18:06:12.798272 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:12.798276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:12.798449 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:12.798607 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:12.799064 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.799836 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.801595 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.802462 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.802588 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.802763 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:12.803626 (+ 863us) server_negotiation.cc:299] Negotiation successful
0810 18:06:12.803738 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":84,"threads_started":1}
I20260810 18:06:12.803279 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.796963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:44373 (local address 127.0.0.1:52600)
0810 18:06:12.798040 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:12.798056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:12.798170 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:12.798671 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:12.798682 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:12.798717 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:12.798938 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.798949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.800013 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.800020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.801448 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.801459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.802611 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.802616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.802742 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.802807 (+ 65us) client_negotiation.cc:769] Sending connection context
0810 18:06:12.802933 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 18:06:12.803086 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":84,"thread_start_us":128,"threads_started":1}
I20260810 18:06:12.804082 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.797259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.21.182.67:36273)
0810 18:06:12.798081 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:12.798090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:12.798110 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:06:12.798214 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:12.798229 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:12.798434 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:12.798694 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:12.799148 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.800979 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.802121 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.802902 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.803012 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.803182 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:12.803803 (+ 621us) server_negotiation.cc:299] Negotiation successful
0810 18:06:12.803931 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":139,"threads_started":1}
I20260810 18:06:12.803469 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.796300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:42603 (local address 127.0.0.1:48072)
0810 18:06:12.797723 (+ 1423us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:12.797737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:12.797805 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:12.798764 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:12.798767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:12.798792 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:12.798963 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.798975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.800790 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.800793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.801890 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.801899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.802943 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.802950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.803051 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.803075 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:06:12.803149 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 18:06:12.803240 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260810 18:06:12.803707 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.796519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:36273 (local address 127.0.0.1:41074)
0810 18:06:12.797563 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:12.797621 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:12.797776 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:12.798864 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:12.798867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:12.798880 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:12.799004 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.799014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.801110 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.801117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.801978 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.801993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.803256 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.803259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.803360 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.803383 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:06:12.803451 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 18:06:12.803574 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":26,"thread_start_us":90,"threads_started":1}
I20260810 18:06:12.807691 23664 tablet_service.cc:1467] Processing CreateTablet for tablet c2063e36289f49b2be69b94ae3fbf337 (DEFAULT_TABLE table=test-table [id=31ccbcb18f07405ebe3fcbadcc3df55c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:12.809105 23664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2063e36289f49b2be69b94ae3fbf337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:12.810186 23827 tablet_service.cc:1467] Processing CreateTablet for tablet c2063e36289f49b2be69b94ae3fbf337 (DEFAULT_TABLE table=test-table [id=31ccbcb18f07405ebe3fcbadcc3df55c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:12.810186 23990 tablet_service.cc:1467] Processing CreateTablet for tablet c2063e36289f49b2be69b94ae3fbf337 (DEFAULT_TABLE table=test-table [id=31ccbcb18f07405ebe3fcbadcc3df55c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:12.811311 23827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2063e36289f49b2be69b94ae3fbf337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:12.811322 23990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2063e36289f49b2be69b94ae3fbf337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:12.816604 24094 tablet_bootstrap.cc:492] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026: Bootstrap starting.
I20260810 18:06:12.820195 24094 tablet_bootstrap.cc:654] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:12.820390 24096 tablet_bootstrap.cc:492] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d: Bootstrap starting.
I20260810 18:06:12.820793 24095 tablet_bootstrap.cc:492] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea: Bootstrap starting.
I20260810 18:06:12.821455 24094 log.cc:824] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:12.822849 24095 tablet_bootstrap.cc:654] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:12.822954 24096 tablet_bootstrap.cc:654] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:12.823628 24095 log.cc:824] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:12.823863 24096 log.cc:824] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:12.824112 24094 tablet_bootstrap.cc:492] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026: No bootstrap required, opened a new log
I20260810 18:06:12.824357 24094 ts_tablet_manager.cc:1397] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 18:06:12.825590 24095 tablet_bootstrap.cc:492] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea: No bootstrap required, opened a new log
I20260810 18:06:12.825791 24095 ts_tablet_manager.cc:1397] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:06:12.827873 24096 tablet_bootstrap.cc:492] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d: No bootstrap required, opened a new log
I20260810 18:06:12.828198 24096 ts_tablet_manager.cc:1397] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260810 18:06:12.829118 24095 raft_consensus.cc:348] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } }
I20260810 18:06:12.829392 24095 raft_consensus.cc:374] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:12.829452 24095 raft_consensus.cc:729] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cace1b79ea497d9842d4caab8b9bea, State: Initialized, Role: FOLLOWER
I20260810 18:06:12.829926 24095 consensus_queue.cc:260] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } }
I20260810 18:06:12.830219 24094 raft_consensus.cc:348] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } }
I20260810 18:06:12.830524 24094 raft_consensus.cc:374] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:12.830600 24094 raft_consensus.cc:729] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fea87ffe1447a892bd572de1957026, State: Initialized, Role: FOLLOWER
I20260810 18:06:12.830869 23901 heartbeater.cc:503] Master 127.21.182.126:44695 was elected leader, sending a full tablet report...
I20260810 18:06:12.831130 24094 consensus_queue.cc:260] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } }
I20260810 18:06:12.832093 23739 heartbeater.cc:503] Master 127.21.182.126:44695 was elected leader, sending a full tablet report...
I20260810 18:06:12.832945 24095 ts_tablet_manager.cc:1428] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:06:12.832871 24096 raft_consensus.cc:348] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } }
I20260810 18:06:12.833256 24096 raft_consensus.cc:374] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:12.832355 24094 ts_tablet_manager.cc:1428] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 18:06:12.833339 24096 raft_consensus.cc:729] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75cf81b973224f4182b13ef439edae8d, State: Initialized, Role: FOLLOWER
I20260810 18:06:12.834370 24096 consensus_queue.cc:260] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } }
I20260810 18:06:12.835644 24067 heartbeater.cc:503] Master 127.21.182.126:44695 was elected leader, sending a full tablet report...
I20260810 18:06:12.835670 24096 ts_tablet_manager.cc:1428] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260810 18:06:12.854234 24102 raft_consensus.cc:482] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:12.854539 24102 raft_consensus.cc:504] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } }
I20260810 18:06:12.855954 24102 leader_election.cc:288] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0cace1b79ea497d9842d4caab8b9bea (127.21.182.66:42603), 75cf81b973224f4182b13ef439edae8d (127.21.182.67:36273)
I20260810 18:06:12.860577 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.856290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:36273 (local address 127.21.182.65:46841)
0810 18:06:12.856733 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:12.856746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:12.856890 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:12.857187 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:12.857193 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:12.857215 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:12.857423 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.857430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.858201 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.858208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.859235 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.859248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.860148 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.860150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.860231 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.860256 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:06:12.860318 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 18:06:12.860375 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":27,"thread_start_us":180,"threads_started":1}
I20260810 18:06:12.860603 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.856493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:46841 (local address 127.21.182.67:36273)
0810 18:06:12.856698 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:12.856704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:12.856825 (+ 121us) server_negotiation.cc:407] Connection header received
0810 18:06:12.857028 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:12.857031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:12.857091 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:12.857150 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:12.857549 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.858081 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.859399 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.860040 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.860164 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.860221 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:12.860365 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 18:06:12.860459 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260810 18:06:12.860946 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.856402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:43891 (local address 127.21.182.66:42603)
0810 18:06:12.856558 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:12.856562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:12.856864 (+ 302us) server_negotiation.cc:407] Connection header received
0810 18:06:12.857076 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:12.857079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:12.857121 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:12.857222 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:12.857577 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.858083 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.859687 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.860359 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.860479 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.860523 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:12.860755 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 18:06:12.860841 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:06:12.861336 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.856306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:42603 (local address 127.21.182.65:43891)
0810 18:06:12.856746 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:12.856761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:12.856957 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:12.857260 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:12.857264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:12.857281 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:12.857437 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.857442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.858463 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.858466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.859567 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:12.859577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.860487 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.860491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:12.860584 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.860620 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 18:06:12.861102 (+ 482us) client_negotiation.cc:240] Negotiation successful
0810 18:06:12.861180 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":54,"spinlock_wait_cycles":20736,"thread_start_us":156,"threads_started":1}
I20260810 18:06:12.861666 24014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2063e36289f49b2be69b94ae3fbf337" candidate_uuid: "64fea87ffe1447a892bd572de1957026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75cf81b973224f4182b13ef439edae8d" is_pre_election: true
I20260810 18:06:12.862046 23848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2063e36289f49b2be69b94ae3fbf337" candidate_uuid: "64fea87ffe1447a892bd572de1957026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cace1b79ea497d9842d4caab8b9bea" is_pre_election: true
I20260810 18:06:12.862152 24014 raft_consensus.cc:2454] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fea87ffe1447a892bd572de1957026 in term 0.
I20260810 18:06:12.862334 23848 raft_consensus.cc:2454] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fea87ffe1447a892bd572de1957026 in term 0.
I20260810 18:06:12.862773 23615 leader_election.cc:302] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 64fea87ffe1447a892bd572de1957026, 75cf81b973224f4182b13ef439edae8d, e0cace1b79ea497d9842d4caab8b9bea; no voters:
I20260810 18:06:12.863111 24102 raft_consensus.cc:2781] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:12.863222 24102 raft_consensus.cc:482] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:12.863323 24102 raft_consensus.cc:3037] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:12.864544 24102 raft_consensus.cc:504] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } }
I20260810 18:06:12.864957 24102 leader_election.cc:288] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [CANDIDATE]: Term 1 election: Requested vote from peers e0cace1b79ea497d9842d4caab8b9bea (127.21.182.66:42603), 75cf81b973224f4182b13ef439edae8d (127.21.182.67:36273)
I20260810 18:06:12.865494 24014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2063e36289f49b2be69b94ae3fbf337" candidate_uuid: "64fea87ffe1447a892bd572de1957026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75cf81b973224f4182b13ef439edae8d"
I20260810 18:06:12.865579 23848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2063e36289f49b2be69b94ae3fbf337" candidate_uuid: "64fea87ffe1447a892bd572de1957026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cace1b79ea497d9842d4caab8b9bea"
I20260810 18:06:12.865670 24014 raft_consensus.cc:3037] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:12.865762 23848 raft_consensus.cc:3037] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:12.866918 23848 raft_consensus.cc:2454] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fea87ffe1447a892bd572de1957026 in term 1.
I20260810 18:06:12.867002 24014 raft_consensus.cc:2454] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fea87ffe1447a892bd572de1957026 in term 1.
I20260810 18:06:12.867363 23616 leader_election.cc:302] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64fea87ffe1447a892bd572de1957026, e0cace1b79ea497d9842d4caab8b9bea; no voters:
I20260810 18:06:12.867656 24102 raft_consensus.cc:2781] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:12.868033 24102 raft_consensus.cc:686] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [term 1 LEADER]: Becoming Leader. State: Replica: 64fea87ffe1447a892bd572de1957026, State: Running, Role: LEADER
I20260810 18:06:12.868506 24102 consensus_queue.cc:237] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } }
I20260810 18:06:12.872584 23511 catalog_manager.cc:5495] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64fea87ffe1447a892bd572de1957026 (127.21.182.65). New cstate: current_term: 1 leader_uuid: "64fea87ffe1447a892bd572de1957026" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } health_report { overall_health: HEALTHY } } }
I20260810 18:06:12.913333 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:12.908945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.21.182.65:44373)
0810 18:06:12.909111 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:12.909115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:12.909132 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:06:12.909293 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:12.909295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:12.909359 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:12.909454 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:12.909801 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.910355 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.911346 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:12.911658 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:12.911817 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:12.912215 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:12.912892 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:12.912995 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:12.913135 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 18:06:12.913200 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:06:12.921108 24014 raft_consensus.cc:1261] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d [term 1 FOLLOWER]: Refusing update from remote peer 64fea87ffe1447a892bd572de1957026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:12.921423 23848 raft_consensus.cc:1261] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea [term 1 FOLLOWER]: Refusing update from remote peer 64fea87ffe1447a892bd572de1957026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:12.922139 24102 consensus_queue.cc:1035] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:12.924449 24102 consensus_queue.cc:1035] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:12.935952 24112 mvcc.cc:205] Tried to move back new op lower bound from 7317033668275486720 to 7317033668072947712. Current Snapshot: MvccSnapshot[applied={T|T < 7317033668275486720}]
I20260810 18:06:12.942132 24110 mvcc.cc:205] Tried to move back new op lower bound from 7317033668275486720 to 7317033668072947712. Current Snapshot: MvccSnapshot[applied={T|T < 7317033668275486720}]
I20260810 18:06:12.958214 23512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:12.958464 23512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:12.962031 23512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35: Sending DeleteTablet for 3 replicas of tablet c2063e36289f49b2be69b94ae3fbf337
I20260810 18:06:12.963289 23664 tablet_service.cc:1514] Processing DeleteTablet for tablet c2063e36289f49b2be69b94ae3fbf337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:12 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:52600
I20260810 18:06:12.963316 23990 tablet_service.cc:1514] Processing DeleteTablet for tablet c2063e36289f49b2be69b94ae3fbf337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:12 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:41074
I20260810 18:06:12.963236 23827 tablet_service.cc:1514] Processing DeleteTablet for tablet c2063e36289f49b2be69b94ae3fbf337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:12 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:48072
I20260810 18:06:12.964077 24119 tablet_replica.cc:330] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026: stopping tablet replica
I20260810 18:06:12.964406 24118 tablet_replica.cc:330] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea: stopping tablet replica
I20260810 18:06:12.964514 24119 raft_consensus.cc:2229] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:06:12.964967 24118 raft_consensus.cc:2229] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:12.965036 24119 raft_consensus.cc:2258] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:12.965612 24118 raft_consensus.cc:2258] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:12.965565 23511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:06:12.966109 23511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:12.966940 24119 ts_tablet_manager.cc:1905] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:12.967880 24120 tablet_replica.cc:330] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d: stopping tablet replica
I20260810 18:06:12.968339 24120 raft_consensus.cc:2229] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:12.968351 24118 ts_tablet_manager.cc:1905] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:12.972492 24118 ts_tablet_manager.cc:1918] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:12.972497 24119 ts_tablet_manager.cc:1918] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:12.972630 24119 log.cc:1196] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/wal/wals/c2063e36289f49b2be69b94ae3fbf337
I20260810 18:06:12.972846 24118 log.cc:1196] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/c2063e36289f49b2be69b94ae3fbf337
I20260810 18:06:12.972998 24119 ts_tablet_manager.cc:1939] T c2063e36289f49b2be69b94ae3fbf337 P 64fea87ffe1447a892bd572de1957026: Deleting consensus metadata
I20260810 18:06:12.973838 24118 ts_tablet_manager.cc:1939] T c2063e36289f49b2be69b94ae3fbf337 P e0cace1b79ea497d9842d4caab8b9bea: Deleting consensus metadata
I20260810 18:06:12.974375 23990 tablet_service.cc:1467] Processing CreateTablet for tablet e2b34bbe57b2416996e5a02d3b2741b7 (DEFAULT_TABLE table=test-table [id=7a717361bf2f46618a9e9c48474f56d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:12.974527 23827 tablet_service.cc:1467] Processing CreateTablet for tablet e2b34bbe57b2416996e5a02d3b2741b7 (DEFAULT_TABLE table=test-table [id=7a717361bf2f46618a9e9c48474f56d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:12.974721 23990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b34bbe57b2416996e5a02d3b2741b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:12.974800 23827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b34bbe57b2416996e5a02d3b2741b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:12.974951 23664 tablet_service.cc:1467] Processing CreateTablet for tablet e2b34bbe57b2416996e5a02d3b2741b7 (DEFAULT_TABLE table=test-table [id=7a717361bf2f46618a9e9c48474f56d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:12.975204 23664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b34bbe57b2416996e5a02d3b2741b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:12.977912 24095 tablet_bootstrap.cc:492] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea: Bootstrap starting.
I20260810 18:06:12.978466 24096 tablet_bootstrap.cc:492] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d: Bootstrap starting.
I20260810 18:06:12.979688 24120 raft_consensus.cc:2258] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:12.980099 24095 tablet_bootstrap.cc:654] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:12.980229 23496 catalog_manager.cc:4841] TS 64fea87ffe1447a892bd572de1957026 (127.21.182.65:44373): tablet c2063e36289f49b2be69b94ae3fbf337 (table test-table [id=31ccbcb18f07405ebe3fcbadcc3df55c]) successfully deleted
I20260810 18:06:12.981218 24096 tablet_bootstrap.cc:654] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:12.981353 24094 tablet_bootstrap.cc:492] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026: Bootstrap starting.
I20260810 18:06:12.982895 24120 ts_tablet_manager.cc:1905] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:12.983083 24094 tablet_bootstrap.cc:654] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:12.983716 24095 tablet_bootstrap.cc:492] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea: No bootstrap required, opened a new log
I20260810 18:06:12.983817 24095 ts_tablet_manager.cc:1397] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260810 18:06:12.985773 24095 raft_consensus.cc:348] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:12.985970 24095 raft_consensus.cc:374] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:12.986008 24095 raft_consensus.cc:729] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cace1b79ea497d9842d4caab8b9bea, State: Initialized, Role: FOLLOWER
I20260810 18:06:12.986177 24095 consensus_queue.cc:260] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:12.986716 23499 catalog_manager.cc:4841] TS e0cace1b79ea497d9842d4caab8b9bea (127.21.182.66:42603): tablet c2063e36289f49b2be69b94ae3fbf337 (table test-table [id=31ccbcb18f07405ebe3fcbadcc3df55c]) successfully deleted
I20260810 18:06:12.986842 24120 ts_tablet_manager.cc:1918] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:12.986956 24120 log.cc:1196] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/c2063e36289f49b2be69b94ae3fbf337
I20260810 18:06:12.987309 24120 ts_tablet_manager.cc:1939] T c2063e36289f49b2be69b94ae3fbf337 P 75cf81b973224f4182b13ef439edae8d: Deleting consensus metadata
I20260810 18:06:12.988328 24094 tablet_bootstrap.cc:492] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026: No bootstrap required, opened a new log
I20260810 18:06:12.988435 24094 ts_tablet_manager.cc:1397] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20260810 18:06:12.988862 23497 catalog_manager.cc:4841] TS 75cf81b973224f4182b13ef439edae8d (127.21.182.67:36273): tablet c2063e36289f49b2be69b94ae3fbf337 (table test-table [id=31ccbcb18f07405ebe3fcbadcc3df55c]) successfully deleted
I20260810 18:06:12.989080 24095 ts_tablet_manager.cc:1428] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260810 18:06:12.990020 24096 tablet_bootstrap.cc:492] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d: No bootstrap required, opened a new log
I20260810 18:06:12.990110 24096 ts_tablet_manager.cc:1397] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20260810 18:06:12.990196 24094 raft_consensus.cc:348] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:12.990327 24094 raft_consensus.cc:374] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:12.990360 24094 raft_consensus.cc:729] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fea87ffe1447a892bd572de1957026, State: Initialized, Role: FOLLOWER
I20260810 18:06:12.990489 24094 consensus_queue.cc:260] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:12.990614 24096 raft_consensus.cc:348] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:12.990731 24094 ts_tablet_manager.cc:1428] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260810 18:06:12.990746 24096 raft_consensus.cc:374] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:12.990789 24096 raft_consensus.cc:729] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75cf81b973224f4182b13ef439edae8d, State: Initialized, Role: FOLLOWER
I20260810 18:06:12.990927 24096 consensus_queue.cc:260] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:12.991200 24096 ts_tablet_manager.cc:1428] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20260810 18:06:12.996897 23741 tablet.cc:2369] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:06:13.087589 24106 raft_consensus.cc:482] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:13.087749 24106 raft_consensus.cc:504] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:13.088398 24106 leader_election.cc:288] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0cace1b79ea497d9842d4caab8b9bea (127.21.182.66:42603), 75cf81b973224f4182b13ef439edae8d (127.21.182.67:36273)
I20260810 18:06:13.088963 23848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b34bbe57b2416996e5a02d3b2741b7" candidate_uuid: "64fea87ffe1447a892bd572de1957026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cace1b79ea497d9842d4caab8b9bea" is_pre_election: true
I20260810 18:06:13.089066 24013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b34bbe57b2416996e5a02d3b2741b7" candidate_uuid: "64fea87ffe1447a892bd572de1957026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75cf81b973224f4182b13ef439edae8d" is_pre_election: true
I20260810 18:06:13.089293 24013 raft_consensus.cc:2454] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fea87ffe1447a892bd572de1957026 in term 0.
I20260810 18:06:13.089301 23848 raft_consensus.cc:2454] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64fea87ffe1447a892bd572de1957026 in term 0.
I20260810 18:06:13.089756 23615 leader_election.cc:302] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 64fea87ffe1447a892bd572de1957026, 75cf81b973224f4182b13ef439edae8d, e0cace1b79ea497d9842d4caab8b9bea; no voters:
I20260810 18:06:13.090054 24106 raft_consensus.cc:2781] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:13.090130 24106 raft_consensus.cc:482] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:13.090202 24106 raft_consensus.cc:3037] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:13.091446 24106 raft_consensus.cc:504] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:13.091861 24106 leader_election.cc:288] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [CANDIDATE]: Term 1 election: Requested vote from peers e0cace1b79ea497d9842d4caab8b9bea (127.21.182.66:42603), 75cf81b973224f4182b13ef439edae8d (127.21.182.67:36273)
I20260810 18:06:13.092314 24013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b34bbe57b2416996e5a02d3b2741b7" candidate_uuid: "64fea87ffe1447a892bd572de1957026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75cf81b973224f4182b13ef439edae8d"
I20260810 18:06:13.092332 23848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b34bbe57b2416996e5a02d3b2741b7" candidate_uuid: "64fea87ffe1447a892bd572de1957026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cace1b79ea497d9842d4caab8b9bea"
I20260810 18:06:13.092464 24013 raft_consensus.cc:3037] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:13.092464 23848 raft_consensus.cc:3037] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:13.093963 23848 raft_consensus.cc:2454] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fea87ffe1447a892bd572de1957026 in term 1.
I20260810 18:06:13.094206 24013 raft_consensus.cc:2454] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64fea87ffe1447a892bd572de1957026 in term 1.
I20260810 18:06:13.094496 23616 leader_election.cc:302] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64fea87ffe1447a892bd572de1957026, e0cace1b79ea497d9842d4caab8b9bea; no voters:
I20260810 18:06:13.094777 24106 raft_consensus.cc:2781] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:13.094995 24106 raft_consensus.cc:686] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [term 1 LEADER]: Becoming Leader. State: Replica: 64fea87ffe1447a892bd572de1957026, State: Running, Role: LEADER
I20260810 18:06:13.095314 24106 consensus_queue.cc:237] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:13.097872 23511 catalog_manager.cc:5495] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64fea87ffe1447a892bd572de1957026 (127.21.182.65). New cstate: current_term: 1 leader_uuid: "64fea87ffe1447a892bd572de1957026" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } health_report { overall_health: UNKNOWN } } }
Aug 10 18:06:13 dist-test-slave-27sr krb5kdc[23464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385171, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:13.157586 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:13.146048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44695 (local address 127.0.0.1:60834)
0810 18:06:13.146541 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:13.146559 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:13.146694 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:13.147011 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:13.147015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:13.147863 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:13.148112 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:13.148122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.149099 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.149103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:13.150339 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:13.150351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.150860 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.150864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:13.150955 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:13.151764 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:13.151780 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:13.153715 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:13.155754 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:13.155766 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:13.155776 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:13.156108 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:13.156419 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:13.156422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:13.156424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:13.156513 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:13.156949 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:13.156957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:13.157077 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 18:06:13.157192 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 18:06:13.157374 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":163,"threads_started":1}
I20260810 18:06:13.157585 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:13.146166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.21.182.126:44695)
0810 18:06:13.146328 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:13.146333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:13.146675 (+ 342us) server_negotiation.cc:407] Connection header received
0810 18:06:13.146790 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:13.146792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:13.146838 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:13.146919 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:13.148275 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.148983 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.150506 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.150721 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.150816 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:13.153905 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:13.153909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:13.153911 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:13.153933 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:13.155630 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:13.156211 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:13.156215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:13.156217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:13.156306 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:13.156608 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:13.156612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:13.156613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:13.156809 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:13.156923 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:13.157258 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 18:06:13.157458 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":65}
I20260810 18:06:13.162972 23511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:60834:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 18:06:13.163861 23511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:13.170725 23664 tablet_service.cc:1467] Processing CreateTablet for tablet b13f454793124806bcacfe6a8f0af311 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bef66fd22fbd41b5a3fbb06e092d1dab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:06:13.170707 23827 tablet_service.cc:1467] Processing CreateTablet for tablet b13f454793124806bcacfe6a8f0af311 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bef66fd22fbd41b5a3fbb06e092d1dab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:06:13.171006 23990 tablet_service.cc:1467] Processing CreateTablet for tablet b13f454793124806bcacfe6a8f0af311 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bef66fd22fbd41b5a3fbb06e092d1dab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:06:13.171265 23664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13f454793124806bcacfe6a8f0af311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:13.171280 23827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13f454793124806bcacfe6a8f0af311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:13.171373 23990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13f454793124806bcacfe6a8f0af311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:13.174863 24096 tablet_bootstrap.cc:492] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d: Bootstrap starting.
I20260810 18:06:13.174880 24094 tablet_bootstrap.cc:492] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026: Bootstrap starting.
I20260810 18:06:13.175982 24094 tablet_bootstrap.cc:654] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:13.176688 24096 tablet_bootstrap.cc:654] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:13.177981 24095 tablet_bootstrap.cc:492] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea: Bootstrap starting.
I20260810 18:06:13.179114 24094 tablet_bootstrap.cc:492] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026: No bootstrap required, opened a new log
I20260810 18:06:13.179250 24094 ts_tablet_manager.cc:1397] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260810 18:06:13.179437 24095 tablet_bootstrap.cc:654] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:13.180425 24094 raft_consensus.cc:348] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:13.180685 24094 raft_consensus.cc:374] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:13.180802 24094 raft_consensus.cc:729] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fea87ffe1447a892bd572de1957026, State: Initialized, Role: FOLLOWER
I20260810 18:06:13.181048 24094 consensus_queue.cc:260] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:13.181552 24106 tablet_replica.cc:439] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } }
I20260810 18:06:13.181654 24095 tablet_bootstrap.cc:492] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea: No bootstrap required, opened a new log
I20260810 18:06:13.181718 24106 tablet_replica.cc:442] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:13.181751 24095 ts_tablet_manager.cc:1397] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260810 18:06:13.182339 24094 ts_tablet_manager.cc:1428] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260810 18:06:13.182605 24095 raft_consensus.cc:348] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:13.182739 24095 raft_consensus.cc:374] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:13.182789 24095 raft_consensus.cc:729] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cace1b79ea497d9842d4caab8b9bea, State: Initialized, Role: FOLLOWER
I20260810 18:06:13.182978 24095 consensus_queue.cc:260] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:13.183365 24101 tablet_replica.cc:439] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } }
I20260810 18:06:13.183559 24101 tablet_replica.cc:442] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:13.184054 24096 tablet_bootstrap.cc:492] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d: No bootstrap required, opened a new log
I20260810 18:06:13.184171 24096 ts_tablet_manager.cc:1397] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20260810 18:06:13.184273 24095 ts_tablet_manager.cc:1428] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 18:06:13.184882 24096 raft_consensus.cc:348] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:13.185007 24096 raft_consensus.cc:374] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:13.185039 24096 raft_consensus.cc:729] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75cf81b973224f4182b13ef439edae8d, State: Initialized, Role: FOLLOWER
I20260810 18:06:13.185170 24096 consensus_queue.cc:260] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:13.185729 24096 ts_tablet_manager.cc:1428] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:06:13.186234 24103 tablet_replica.cc:439] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } }
I20260810 18:06:13.186421 24103 tablet_replica.cc:442] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 18:06:13.230484 24068 tablet.cc:2369] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:06:13.233135 23902 tablet.cc:2369] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:06:13.275940 24103 raft_consensus.cc:482] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:13.276109 24103 raft_consensus.cc:504] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:13.277451 24103 leader_election.cc:288] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64fea87ffe1447a892bd572de1957026 (127.21.182.65:44373), e0cace1b79ea497d9842d4caab8b9bea (127.21.182.66:42603)
I20260810 18:06:13.283753 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:13.278025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:56293 (local address 127.21.182.66:42603)
0810 18:06:13.278161 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:13.278167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:13.278379 (+ 212us) server_negotiation.cc:407] Connection header received
0810 18:06:13.278495 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:13.278499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:13.278557 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:13.278671 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:13.279220 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.279981 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.282202 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.282808 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.282959 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:13.283026 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:13.283507 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 18:06:13.283595 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:06:13.283814 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:13.278016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:53991 (local address 127.21.182.65:44373)
0810 18:06:13.278162 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:13.278167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:13.278379 (+ 212us) server_negotiation.cc:407] Connection header received
0810 18:06:13.278495 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:13.278499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:13.278557 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:13.278770 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:13.279088 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.280011 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.282180 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.282896 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.283014 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:13.283074 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:13.283507 (+ 433us) server_negotiation.cc:299] Negotiation successful
0810 18:06:13.283599 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:06:13.283876 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:13.277812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:44373 (local address 127.21.182.67:53991)
0810 18:06:13.278213 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:13.278230 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:13.278389 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:13.278715 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:13.278719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:13.278746 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:13.278889 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:13.278897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.280156 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.280161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:13.282012 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:13.282027 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.283019 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.283022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:13.283320 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:13.283376 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 18:06:13.283465 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 18:06:13.283606 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":287,"spinlock_wait_cycles":29952,"thread_start_us":92,"threads_started":1}
I20260810 18:06:13.283973 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:13.277833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:42603 (local address 127.21.182.67:56293)
0810 18:06:13.278253 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:13.278269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:13.278389 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:13.278775 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:13.278777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:13.278799 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:13.279062 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:13.279072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.280156 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.280161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:13.282012 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:13.282027 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.282984 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.282990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:13.283308 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:13.283370 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 18:06:13.283465 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 18:06:13.283603 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":161,"spinlock_wait_cycles":12672,"thread_start_us":91,"threads_started":1}
I20260810 18:06:13.284564 23848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13f454793124806bcacfe6a8f0af311" candidate_uuid: "75cf81b973224f4182b13ef439edae8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cace1b79ea497d9842d4caab8b9bea" is_pre_election: true
I20260810 18:06:13.284608 23685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13f454793124806bcacfe6a8f0af311" candidate_uuid: "75cf81b973224f4182b13ef439edae8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fea87ffe1447a892bd572de1957026" is_pre_election: true
I20260810 18:06:13.284747 23848 raft_consensus.cc:2454] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75cf81b973224f4182b13ef439edae8d in term 0.
I20260810 18:06:13.284942 23685 raft_consensus.cc:2454] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75cf81b973224f4182b13ef439edae8d in term 0.
I20260810 18:06:13.285225 23937 leader_election.cc:302] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75cf81b973224f4182b13ef439edae8d, e0cace1b79ea497d9842d4caab8b9bea; no voters:
I20260810 18:06:13.285532 24103 raft_consensus.cc:2781] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:13.285600 24103 raft_consensus.cc:482] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:13.285640 24103 raft_consensus.cc:3037] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:13.287066 24103 raft_consensus.cc:504] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:13.287434 24103 leader_election.cc:288] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [CANDIDATE]: Term 1 election: Requested vote from peers 64fea87ffe1447a892bd572de1957026 (127.21.182.65:44373), e0cace1b79ea497d9842d4caab8b9bea (127.21.182.66:42603)
I20260810 18:06:13.287811 23848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13f454793124806bcacfe6a8f0af311" candidate_uuid: "75cf81b973224f4182b13ef439edae8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cace1b79ea497d9842d4caab8b9bea"
I20260810 18:06:13.287861 23685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13f454793124806bcacfe6a8f0af311" candidate_uuid: "75cf81b973224f4182b13ef439edae8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fea87ffe1447a892bd572de1957026"
I20260810 18:06:13.288023 23848 raft_consensus.cc:3037] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:13.288060 23685 raft_consensus.cc:3037] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:13.289304 23848 raft_consensus.cc:2454] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75cf81b973224f4182b13ef439edae8d in term 1.
I20260810 18:06:13.289455 23685 raft_consensus.cc:2454] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75cf81b973224f4182b13ef439edae8d in term 1.
I20260810 18:06:13.290421 23935 leader_election.cc:302] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64fea87ffe1447a892bd572de1957026, 75cf81b973224f4182b13ef439edae8d; no voters:
I20260810 18:06:13.290723 24103 raft_consensus.cc:2781] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:13.290920 24103 raft_consensus.cc:686] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [term 1 LEADER]: Becoming Leader. State: Replica: 75cf81b973224f4182b13ef439edae8d, State: Running, Role: LEADER
I20260810 18:06:13.291213 24103 consensus_queue.cc:237] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } }
I20260810 18:06:13.292804 24140 tablet_replica.cc:439] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d: TxnStatusTablet state changed. Reason: New leader 75cf81b973224f4182b13ef439edae8d. Latest consensus state: current_term: 1 leader_uuid: "75cf81b973224f4182b13ef439edae8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } }
I20260810 18:06:13.292945 24140 tablet_replica.cc:442] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:06:13.293404 24142 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:06:13.293648 24142 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 18:06:13.294219 23511 catalog_manager.cc:5495] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 75cf81b973224f4182b13ef439edae8d (127.21.182.67). New cstate: current_term: 1 leader_uuid: "75cf81b973224f4182b13ef439edae8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } health_report { overall_health: HEALTHY } } }
I20260810 18:06:13.317139 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:13.312136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:36273 (local address 127.0.0.1:41090)
0810 18:06:13.312287 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:13.312323 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:13.312446 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:13.312898 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:13.312901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:13.312913 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:06:13.313061 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:13.313068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.314363 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.314366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:13.315037 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:13.315047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.315747 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.315752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:13.315830 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:13.315884 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:06:13.316856 (+ 972us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:06:13.316872 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 18:06:13.316933 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 18:06:13.317011 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 18:06:13.317178 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:13.312242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.21.182.67:36273)
0810 18:06:13.312374 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:13.312378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:13.312397 (+ 19us) server_negotiation.cc:407] Connection header received
0810 18:06:13.312608 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:13.312611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:13.312735 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:13.312853 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:13.313205 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.314192 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.315172 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.315578 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.315723 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:13.316094 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:13.316751 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:13.316874 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:13.316946 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 18:06:13.317006 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:06:13.320896 23848 raft_consensus.cc:1261] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea [term 1 FOLLOWER]: Refusing update from remote peer 75cf81b973224f4182b13ef439edae8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:13.321075 23685 raft_consensus.cc:1261] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026 [term 1 FOLLOWER]: Refusing update from remote peer 75cf81b973224f4182b13ef439edae8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:13.321563 24103 consensus_queue.cc:1035] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:13.321772 24140 consensus_queue.cc:1035] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:13.323575 24101 tablet_replica.cc:439] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea: TxnStatusTablet state changed. Reason: New leader 75cf81b973224f4182b13ef439edae8d. Latest consensus state: current_term: 1 leader_uuid: "75cf81b973224f4182b13ef439edae8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } }
I20260810 18:06:13.323717 24101 tablet_replica.cc:442] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:13.323999 24106 tablet_replica.cc:439] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026: TxnStatusTablet state changed. Reason: New leader 75cf81b973224f4182b13ef439edae8d. Latest consensus state: current_term: 1 leader_uuid: "75cf81b973224f4182b13ef439edae8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } }
I20260810 18:06:13.324141 24106 tablet_replica.cc:442] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:13.326165 24140 tablet_replica.cc:439] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75cf81b973224f4182b13ef439edae8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } }
I20260810 18:06:13.326313 24140 tablet_replica.cc:442] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:06:13.326929 24101 tablet_replica.cc:439] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75cf81b973224f4182b13ef439edae8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } }
I20260810 18:06:13.327042 24101 tablet_replica.cc:442] T b13f454793124806bcacfe6a8f0af311 P e0cace1b79ea497d9842d4caab8b9bea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:13.328045 24103 tablet_replica.cc:439] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75cf81b973224f4182b13ef439edae8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } }
I20260810 18:06:13.328157 24103 tablet_replica.cc:442] T b13f454793124806bcacfe6a8f0af311 P 75cf81b973224f4182b13ef439edae8d: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:06:13.329499 24109 mvcc.cc:205] Tried to move back new op lower bound from 7317033669916385280 to 7317033669803823104. Current Snapshot: MvccSnapshot[applied={T|T < 7317033669916385280}]
I20260810 18:06:13.329696 24106 tablet_replica.cc:439] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75cf81b973224f4182b13ef439edae8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fea87ffe1447a892bd572de1957026" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44373 } } peers { permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 } } peers { permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 } } }
I20260810 18:06:13.329802 24106 tablet_replica.cc:442] T b13f454793124806bcacfe6a8f0af311 P 64fea87ffe1447a892bd572de1957026: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:13.345191 24151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:13.340477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:36273 (local address 127.21.182.65:59101)
0810 18:06:13.341189 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:13.341205 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:13.341319 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:13.341627 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:13.341629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:13.341638 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:06:13.341752 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:13.341757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.342952 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.342955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:13.343631 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:13.343641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.344417 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.344420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:13.344494 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:13.344551 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:06:13.344898 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:06:13.344904 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:06:13.344968 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 18:06:13.345051 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":283,"threads_started":1}
I20260810 18:06:13.345264 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:13.340664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:59101 (local address 127.21.182.67:36273)
0810 18:06:13.340850 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:13.340856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:13.341301 (+ 445us) server_negotiation.cc:407] Connection header received
0810 18:06:13.341436 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:13.341440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:13.341505 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:13.341605 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:13.341876 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.342802 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.343821 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.344285 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.344413 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:13.344701 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:13.344803 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:13.344884 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:13.345010 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 18:06:13.345104 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 18:06:13.353961 23848 raft_consensus.cc:1261] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea [term 1 FOLLOWER]: Refusing update from remote peer 64fea87ffe1447a892bd572de1957026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:13.354084 24013 raft_consensus.cc:1261] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d [term 1 FOLLOWER]: Refusing update from remote peer 64fea87ffe1447a892bd572de1957026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:13.354564 24106 consensus_queue.cc:1035] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75cf81b973224f4182b13ef439edae8d" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 36273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:13.354878 24102 consensus_queue.cc:1035] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cace1b79ea497d9842d4caab8b9bea" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 42603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:13.384548 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:13.379434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:46681 (local address 127.21.182.65:44373)
0810 18:06:13.379574 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:13.379578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:13.379686 (+ 108us) server_negotiation.cc:407] Connection header received
0810 18:06:13.379846 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:13.379848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:13.379953 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:13.380030 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:13.380612 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.381142 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.383099 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.383369 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.383461 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:13.383972 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:13.384075 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:13.384176 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:13.384291 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 18:06:13.384355 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:06:13.384687 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:13.378946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:44373 (local address 127.21.182.67:46681)
0810 18:06:13.379607 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:13.379626 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:13.379754 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:13.380070 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:13.380074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:13.380083 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:06:13.380404 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:13.380412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.381629 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.381632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:13.382957 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:13.382971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.383494 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.383498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:13.383678 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:13.383752 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:06:13.384183 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:06:13.384193 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 18:06:13.384260 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 18:06:13.384357 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":169,"threads_started":1}
I20260810 18:06:13.411255 23511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:13.411454 23511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:13.414152 23511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82d45692ca63498db7775be835977f35: Sending DeleteTablet for 3 replicas of tablet e2b34bbe57b2416996e5a02d3b2741b7
I20260810 18:06:13.414968 23827 tablet_service.cc:1514] Processing DeleteTablet for tablet e2b34bbe57b2416996e5a02d3b2741b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:13 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:48072
I20260810 18:06:13.415237 24118 tablet_replica.cc:330] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea: stopping tablet replica
I20260810 18:06:13.415377 24118 raft_consensus.cc:2229] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:13.415515 24118 raft_consensus.cc:2258] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:13.415717 23664 tablet_service.cc:1514] Processing DeleteTablet for tablet e2b34bbe57b2416996e5a02d3b2741b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:13 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:52600
I20260810 18:06:13.415992 24119 tablet_replica.cc:330] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026: stopping tablet replica
I20260810 18:06:13.416049 23990 tablet_service.cc:1514] Processing DeleteTablet for tablet e2b34bbe57b2416996e5a02d3b2741b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:13 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:41074
I20260810 18:06:13.416136 24119 raft_consensus.cc:2229] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:06:13.416357 24120 tablet_replica.cc:330] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d: stopping tablet replica
I20260810 18:06:13.416496 24120 raft_consensus.cc:2229] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:13.416508 24119 raft_consensus.cc:2258] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:13.416651 24120 raft_consensus.cc:2258] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:13.417446 24118 ts_tablet_manager.cc:1905] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:13.417704 24119 ts_tablet_manager.cc:1905] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:13.417853 24120 ts_tablet_manager.cc:1905] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:13.420751 24120 ts_tablet_manager.cc:1918] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:06:13.420861 24120 log.cc:1196] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/e2b34bbe57b2416996e5a02d3b2741b7
I20260810 18:06:13.420858 24118 ts_tablet_manager.cc:1918] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:06:13.420858 24119 ts_tablet_manager.cc:1918] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:06:13.420940 24119 log.cc:1196] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-0/wal/wals/e2b34bbe57b2416996e5a02d3b2741b7
I20260810 18:06:13.420940 24118 log.cc:1196] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/e2b34bbe57b2416996e5a02d3b2741b7
I20260810 18:06:13.421151 24120 ts_tablet_manager.cc:1939] T e2b34bbe57b2416996e5a02d3b2741b7 P 75cf81b973224f4182b13ef439edae8d: Deleting consensus metadata
I20260810 18:06:13.421238 24119 ts_tablet_manager.cc:1939] T e2b34bbe57b2416996e5a02d3b2741b7 P 64fea87ffe1447a892bd572de1957026: Deleting consensus metadata
I20260810 18:06:13.421268 24118 ts_tablet_manager.cc:1939] T e2b34bbe57b2416996e5a02d3b2741b7 P e0cace1b79ea497d9842d4caab8b9bea: Deleting consensus metadata
I20260810 18:06:13.422097 23497 catalog_manager.cc:4841] TS 75cf81b973224f4182b13ef439edae8d (127.21.182.67:36273): tablet e2b34bbe57b2416996e5a02d3b2741b7 (table test-table [id=7a717361bf2f46618a9e9c48474f56d8]) successfully deleted
I20260810 18:06:13.422293 23499 catalog_manager.cc:4841] TS e0cace1b79ea497d9842d4caab8b9bea (127.21.182.66:42603): tablet e2b34bbe57b2416996e5a02d3b2741b7 (table test-table [id=7a717361bf2f46618a9e9c48474f56d8]) successfully deleted
I20260810 18:06:13.422663 23496 catalog_manager.cc:4841] TS 64fea87ffe1447a892bd572de1957026 (127.21.182.65:44373): tablet e2b34bbe57b2416996e5a02d3b2741b7 (table test-table [id=7a717361bf2f46618a9e9c48474f56d8]) successfully deleted
Aug 10 18:06:13 dist-test-slave-27sr krb5kdc[23464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.182.65@KRBTEST.COM
I20260810 18:06:13.434415 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:13.416714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.21.182.65:44373)
0810 18:06:13.416842 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:13.416849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:13.424243 (+ 7394us) server_negotiation.cc:407] Connection header received
0810 18:06:13.424424 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:13.424427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:13.424489 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:13.424580 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:13.425308 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.425839 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.426690 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.426944 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.427062 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:13.430045 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:13.430059 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:13.430077 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:13.430109 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:13.432319 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:13.432873 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:13.432879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:13.432881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:13.432976 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:13.433347 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:13.433350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:13.433352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:13.433658 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:13.433914 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:13.434091 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 18:06:13.434241 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 18:06:13.435132 23733 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52618
I20260810 18:06:13.446504 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:13.437177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.21.182.126:44695)
0810 18:06:13.437368 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:13.437372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:13.437387 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:06:13.437514 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:13.437516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:13.437560 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:13.437629 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:13.438545 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.439136 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.440463 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:13.440686 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:13.440787 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:13.442506 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:13.442510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:13.442512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:13.442545 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:13.444468 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:13.444973 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:13.444976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:13.444977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:13.445074 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:13.445416 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:13.445420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:13.445421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:13.445713 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:13.445846 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:13.446179 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 18:06:13.446295 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260810 18:06:13.446988 23511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60836
I20260810 18:06:13.448360 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 23596
I20260810 18:06:13.468732 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 23750
I20260810 18:06:13.482316 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 23913
I20260810 18:06:13.499567 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 23478
2026-08-10T18:06:13Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3122 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:06:13 dist-test-slave-27sr krb5kdc[24167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:06:13 dist-test-slave-27sr krb5kdc[24167](info): set up 2 sockets
Aug 10 18:06:13 dist-test-slave-27sr krb5kdc[24167](info): commencing operation
krb5kdc: starting...
W20260810 18:06:14.723563 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:06:14 dist-test-slave-27sr krb5kdc[24167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:06:14.867965 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:46085
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:46085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:14.994844 24181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:14.995253 24181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:14.995304 24181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:14.999473 24181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:14.999548 24181 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 18:06:14.999567 24181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:14.999588 24181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:14.999603 24181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:14.999619 24181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:15.005494 24181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:46085
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:46085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:15.006649 24181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:15.007688 24181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:15.017167 24189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:15.017160 24187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:15.017279 24181 server_base.cc:1034] running on GCE node
W20260810 18:06:15.017160 24186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:15.018229 24181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:15.019542 24181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:15.020747 24181 hybrid_clock.cc:648] HybridClock initialized: now 1786385175020711 us; error 51 us; skew 500 ppm
Aug 10 18:06:15 dist-test-slave-27sr krb5kdc[24167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385175, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:15.023800 24181 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:06:15.024816 24181 webserver.cc:460] Webserver started at http://127.21.182.126:43057/ using document root <none> and password file <none>
I20260810 18:06:15.025394 24181 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:15.025453 24181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:15.025632 24181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:15.027772 24181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "1e1c23de1ddb467cb2f765742a1dcdbf"
format_stamp: "Formatted at 2026-08-10 18:06:15 on dist-test-slave-27sr"
server_key: "8d3824ec073fe97df6e19fc33009bf90"
server_key_iv: "281ee43877b148f8ddbb9c2e7788c437"
server_key_version: "encryptionkey@0"
I20260810 18:06:15.028469 24181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "1e1c23de1ddb467cb2f765742a1dcdbf"
format_stamp: "Formatted at 2026-08-10 18:06:15 on dist-test-slave-27sr"
server_key: "8d3824ec073fe97df6e19fc33009bf90"
server_key_iv: "281ee43877b148f8ddbb9c2e7788c437"
server_key_version: "encryptionkey@0"
I20260810 18:06:15.032552 24181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:06:15.035048 24197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:15.036607 24181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:15.036754 24181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "1e1c23de1ddb467cb2f765742a1dcdbf"
format_stamp: "Formatted at 2026-08-10 18:06:15 on dist-test-slave-27sr"
server_key: "8d3824ec073fe97df6e19fc33009bf90"
server_key_iv: "281ee43877b148f8ddbb9c2e7788c437"
server_key_version: "encryptionkey@0"
I20260810 18:06:15.036872 24181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:15.065480 24181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:15.069430 24181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:15.069661 24181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:15.079022 24181 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:46085
I20260810 18:06:15.079039 24269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:46085 every 8 connection(s)
I20260810 18:06:15.080219 24181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:15.083412 24270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:15.087255 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 24181
I20260810 18:06:15.087430 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:15.087702 22233 external_mini_cluster.cc:1428] Setting key a7120ec62d15c357dccbb5e91a2395ba
I20260810 18:06:15.090113 24270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf: Bootstrap starting.
I20260810 18:06:15.093035 24270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:15.093981 24270 log.cc:824] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:15.096712 24270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf: No bootstrap required, opened a new log
Aug 10 18:06:15 dist-test-slave-27sr krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:15.101411 24270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1c23de1ddb467cb2f765742a1dcdbf" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 46085 } }
I20260810 18:06:15.101682 24270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:15.101788 24270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e1c23de1ddb467cb2f765742a1dcdbf, State: Initialized, Role: FOLLOWER
I20260810 18:06:15.102355 24270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1c23de1ddb467cb2f765742a1dcdbf" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 46085 } }
I20260810 18:06:15.102499 24270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:15.102589 24270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:15.102705 24270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:15.103824 24270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1c23de1ddb467cb2f765742a1dcdbf" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 46085 } }
I20260810 18:06:15.104256 24270 leader_election.cc:302] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e1c23de1ddb467cb2f765742a1dcdbf; no voters:
I20260810 18:06:15.104645 24270 leader_election.cc:288] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:15.104753 24275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:15.104789 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.089336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.21.182.126:46085)
0810 18:06:15.089849 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:15.089866 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:15.089904 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:06:15.090610 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:15.090630 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:15.090986 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:15.091274 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:15.092358 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.093497 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.094729 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.095271 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.095406 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.098948 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:15.098954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:15.098971 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:15.099004 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:15.101116 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.101644 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.101668 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.101678 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.101895 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.102224 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.102228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.102230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.102851 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:15.103101 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:15.103449 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 18:06:15.103788 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":159,"threads_started":1}
I20260810 18:06:15.104971 24275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [term 1 LEADER]: Becoming Leader. State: Replica: 1e1c23de1ddb467cb2f765742a1dcdbf, State: Running, Role: LEADER
I20260810 18:06:15.105309 24275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1c23de1ddb467cb2f765742a1dcdbf" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 46085 } }
I20260810 18:06:15.106238 24270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:15.106710 24275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e1c23de1ddb467cb2f765742a1dcdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1c23de1ddb467cb2f765742a1dcdbf" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 46085 } } }
I20260810 18:06:15.106841 24275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:15.107064 24275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e1c23de1ddb467cb2f765742a1dcdbf. Latest consensus state: current_term: 1 leader_uuid: "1e1c23de1ddb467cb2f765742a1dcdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1c23de1ddb467cb2f765742a1dcdbf" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 46085 } } }
I20260810 18:06:15.107188 24275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:15.108254 24279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:15.111444 24279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:15.118291 24279 catalog_manager.cc:1347] Generated new cluster ID: fba28998d6ce4c93a20539ce8bcbb712
I20260810 18:06:15.118402 24279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:15.133064 24279 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:15.134259 24279 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:15.140938 24279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf: Generated new TSK 0
I20260810 18:06:15.141877 24279 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:06:15.180190 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:46085
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:15.299160 24298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:15.299479 24298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:15.299574 24298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:15.303645 24298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:15.303752 24298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:15.303833 24298 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 18:06:15.303876 24298 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:06:15.303959 24298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:15.309701 24298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.182.126:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:15.310842 24298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:15.311916 24298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:15.322088 24306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:15.322178 24298 server_base.cc:1034] running on GCE node
W20260810 18:06:15.322103 24303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:15.322350 24304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:15.322782 24298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:15.323447 24298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:15.324622 24298 hybrid_clock.cc:648] HybridClock initialized: now 1786385175324592 us; error 40 us; skew 500 ppm
Aug 10 18:06:15 dist-test-slave-27sr krb5kdc[24167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385175, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:15.333570 24298 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:06:15.334904 24298 webserver.cc:460] Webserver started at http://127.21.182.65:41727/ using document root <none> and password file <none>
I20260810 18:06:15.335606 24298 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:15.335721 24298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:15.336063 24298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:15.338547 24298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "41b3f230abf543aab1f0567731bbd5bb"
format_stamp: "Formatted at 2026-08-10 18:06:15 on dist-test-slave-27sr"
server_key: "33721cb246d029399b4247a445ea859a"
server_key_iv: "d7408886bbece210f0aba31ffa6fa1c0"
server_key_version: "encryptionkey@0"
I20260810 18:06:15.339277 24298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "41b3f230abf543aab1f0567731bbd5bb"
format_stamp: "Formatted at 2026-08-10 18:06:15 on dist-test-slave-27sr"
server_key: "33721cb246d029399b4247a445ea859a"
server_key_iv: "d7408886bbece210f0aba31ffa6fa1c0"
server_key_version: "encryptionkey@0"
I20260810 18:06:15.344008 24298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.008s
I20260810 18:06:15.346671 24313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:15.348099 24298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 18:06:15.348253 24298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "41b3f230abf543aab1f0567731bbd5bb"
format_stamp: "Formatted at 2026-08-10 18:06:15 on dist-test-slave-27sr"
server_key: "33721cb246d029399b4247a445ea859a"
server_key_iv: "d7408886bbece210f0aba31ffa6fa1c0"
server_key_version: "encryptionkey@0"
I20260810 18:06:15.348376 24298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:15.370383 24298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:15.374044 24298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:15.374284 24298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:15.376056 24298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:15.376133 24298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:15.376199 24298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:15.376257 24298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:06:15 dist-test-slave-27sr krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385175, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:15.387877 24298 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:45337
I20260810 18:06:15.388374 24438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:45337 every 8 connection(s)
I20260810 18:06:15.388929 24298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:15.391383 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.377091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:52813 (local address 127.21.182.126:46085)
0810 18:06:15.377279 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:15.377285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:15.378284 (+ 999us) server_negotiation.cc:407] Connection header received
0810 18:06:15.379660 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:15.379665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:15.379731 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:15.379850 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:15.381512 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.382200 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.383476 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.383875 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.384046 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.387316 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:15.387322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:15.387328 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:15.387364 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:15.389128 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.389804 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.389806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.389808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.389884 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.390346 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.390348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.390349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.390525 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:15.390605 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:15.391148 (+ 543us) server_negotiation.cc:299] Negotiation successful
0810 18:06:15.391255 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 18:06:15.392407 24323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.377515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:46085 (local address 127.21.182.65:52813)
0810 18:06:15.378098 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:15.378160 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:15.379391 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:15.380067 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:15.380079 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:15.380571 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:15.381344 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.381371 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.382350 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.382354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.383329 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.383340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.384106 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.384111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.384360 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.385009 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:15.385037 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:15.387108 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:15.389258 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:15.389268 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:15.389285 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:15.389658 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:15.390059 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:15.390062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:15.390063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:15.390242 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:15.390651 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:15.390657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:15.390922 (+ 265us) client_negotiation.cc:769] Sending connection context
0810 18:06:15.391101 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 18:06:15.391365 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":126,"threads_started":1}
I20260810 18:06:15.394060 24439 heartbeater.cc:347] Connected to a master server at 127.21.182.126:46085
I20260810 18:06:15.394315 24439 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:15.394951 24439 heartbeater.cc:511] Master 127.21.182.126:46085 requested a full tablet report, sending...
I20260810 18:06:15.396885 24217 ts_manager.cc:194] Registered new tserver with Master: 41b3f230abf543aab1f0567731bbd5bb (127.21.182.65:45337)
I20260810 18:06:15.398499 24217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:52813
I20260810 18:06:15.398790 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 24298
I20260810 18:06:15.398919 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:06:15.399274 22233 external_mini_cluster.cc:1428] Setting key 195836986cfa0313b1686d8e6fc0afb0
I20260810 18:06:15.405567 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.396865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:43837 (local address 127.21.182.126:46085)
0810 18:06:15.397014 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:15.397019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:15.397291 (+ 272us) server_negotiation.cc:407] Connection header received
0810 18:06:15.397400 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:15.397402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:15.397442 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:15.397506 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:15.398087 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.398765 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.400057 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.400378 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.400493 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.401929 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:15.401935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:15.401940 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:15.401973 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:15.403543 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.404191 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.404194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.404195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.404286 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.404655 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.404659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.404661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.404867 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:15.404959 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:15.405323 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 18:06:15.405439 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:06:15.405843 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.396744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:46085 (local address 127.21.182.65:43837)
0810 18:06:15.397190 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:15.397206 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:15.397315 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:15.397554 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:15.397557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:15.397810 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:15.397937 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.397946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.398941 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.398944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.399858 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.399869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.400744 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.400747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.400829 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.401400 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:15.401413 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:15.401775 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:15.403669 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:15.403675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:15.403678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:15.404029 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:15.404399 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:15.404406 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:15.404411 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:15.404531 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:15.404981 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:15.404988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:15.405114 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 18:06:15.405250 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 18:06:15.405438 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:06:15.431821 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:46085
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:15.552208 24451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:15.552564 24451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:15.552613 24451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:15.556788 24451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:15.556869 24451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:15.556910 24451 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 18:06:15.556926 24451 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:06:15.556958 24451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:15.562354 24451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.182.126:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:15.563472 24451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:15.564567 24451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:15.574898 24461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:15.574988 24458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:15.574919 24459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:15.575220 24451 server_base.cc:1034] running on GCE node
I20260810 18:06:15.575690 24451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:15.576481 24451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:15.577700 24451 hybrid_clock.cc:648] HybridClock initialized: now 1786385175577676 us; error 31 us; skew 500 ppm
Aug 10 18:06:15 dist-test-slave-27sr krb5kdc[24167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385175, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:15.580791 24451 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:06:15.581822 24451 webserver.cc:460] Webserver started at http://127.21.182.66:41927/ using document root <none> and password file <none>
I20260810 18:06:15.582482 24451 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:15.582582 24451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:15.582842 24451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:15.585153 24451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "a614cf9a15b343208e90f06c52901d44"
format_stamp: "Formatted at 2026-08-10 18:06:15 on dist-test-slave-27sr"
server_key: "e8cae49ef64325f52f488baa02c10d50"
server_key_iv: "caf471afc535f874d4dd83ee8b16fbda"
server_key_version: "encryptionkey@0"
I20260810 18:06:15.585784 24451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "a614cf9a15b343208e90f06c52901d44"
format_stamp: "Formatted at 2026-08-10 18:06:15 on dist-test-slave-27sr"
server_key: "e8cae49ef64325f52f488baa02c10d50"
server_key_iv: "caf471afc535f874d4dd83ee8b16fbda"
server_key_version: "encryptionkey@0"
I20260810 18:06:15.589865 24451 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:06:15.592501 24469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:15.593801 24451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:06:15.593971 24451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "a614cf9a15b343208e90f06c52901d44"
format_stamp: "Formatted at 2026-08-10 18:06:15 on dist-test-slave-27sr"
server_key: "e8cae49ef64325f52f488baa02c10d50"
server_key_iv: "caf471afc535f874d4dd83ee8b16fbda"
server_key_version: "encryptionkey@0"
I20260810 18:06:15.594092 24451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:15.613678 24451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:15.617373 24451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:15.617585 24451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:15.619093 24451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:15.619172 24451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:15.619242 24451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:15.619279 24451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:06:15 dist-test-slave-27sr krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385175, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:15.630520 24451 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:39377
I20260810 18:06:15.630630 24595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:39377 every 8 connection(s)
I20260810 18:06:15.631727 24451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:06:15.634313 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.620035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:38543 (local address 127.21.182.126:46085)
0810 18:06:15.620214 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:15.620222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:15.621133 (+ 911us) server_negotiation.cc:407] Connection header received
0810 18:06:15.622260 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:15.622264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:15.622332 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:15.622409 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:15.624270 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.624969 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.626920 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.627252 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.627343 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.630420 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:15.630426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:15.630432 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:15.630470 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:15.632121 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.632757 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.632761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.632764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.632847 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.633197 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.633200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.633202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.633391 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:15.633501 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:15.634011 (+ 510us) server_negotiation.cc:299] Negotiation successful
0810 18:06:15.634135 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:06:15.635206 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.620453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:46085 (local address 127.21.182.66:38543)
0810 18:06:15.620962 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:15.621014 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:15.621989 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:15.622902 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:15.622910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:15.623373 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:15.624082 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.624099 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.625125 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.625132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.626771 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.626788 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.627424 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.627429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.627640 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.628291 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:15.628318 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:15.630226 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:15.632272 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:15.632279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:15.632293 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:15.632623 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:15.632943 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:15.632947 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:15.632949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:15.633089 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:15.633500 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:15.633507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:15.633745 (+ 238us) client_negotiation.cc:769] Sending connection context
0810 18:06:15.633948 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 18:06:15.634263 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":126,"threads_started":1}
I20260810 18:06:15.636592 24598 heartbeater.cc:347] Connected to a master server at 127.21.182.126:46085
I20260810 18:06:15.636787 24598 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:15.637341 24598 heartbeater.cc:511] Master 127.21.182.126:46085 requested a full tablet report, sending...
I20260810 18:06:15.638533 24216 ts_manager.cc:194] Registered new tserver with Master: a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377)
I20260810 18:06:15.639325 24216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:38543
I20260810 18:06:15.639770 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 24451
I20260810 18:06:15.639931 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:06:15.640322 22233 external_mini_cluster.cc:1428] Setting key c2e0ceb4dc690fdf0562a18028eb277a
I20260810 18:06:15.647418 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.639015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:50071 (local address 127.21.182.126:46085)
0810 18:06:15.639156 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:15.639160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:15.639411 (+ 251us) server_negotiation.cc:407] Connection header received
0810 18:06:15.639586 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:15.639589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:15.639648 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:15.639721 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:15.640407 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.641172 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.642382 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.642641 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.642751 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.644017 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:15.644021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:15.644023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:15.644054 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:15.645581 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.646087 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.646092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.646093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.646177 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.646496 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.646499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.646501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.646737 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:15.646851 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:15.647126 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 18:06:15.647247 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:06:15.647432 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.638942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:46085 (local address 127.21.182.66:50071)
0810 18:06:15.639312 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:15.639324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:15.639434 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:15.639922 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:15.639926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:15.640135 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:15.640282 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.640288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.641634 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.641637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.642248 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.642257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.642766 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.642770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.642865 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.643465 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:15.643478 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:15.643826 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:15.645756 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:15.645759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:15.645761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:15.645976 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:15.646281 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:15.646284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:15.646286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:15.646391 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:15.646841 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:15.646844 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:15.646950 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 18:06:15.647065 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 18:06:15.647237 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:06:15.672986 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:46085
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:15.792055 24607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:15.792328 24607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:15.792414 24607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:15.796697 24607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:15.796767 24607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:15.796811 24607 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 18:06:15.796868 24607 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:06:15.796933 24607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:15.802610 24607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.182.126:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:15.803725 24607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:15.804808 24607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:15.814942 24614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:15.814930 24613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:15.814930 24616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:15.815423 24607 server_base.cc:1034] running on GCE node
I20260810 18:06:15.815836 24607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:15.816553 24607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:15.817724 24607 hybrid_clock.cc:648] HybridClock initialized: now 1786385175817699 us; error 35 us; skew 500 ppm
Aug 10 18:06:15 dist-test-slave-27sr krb5kdc[24167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385175, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:15.820890 24607 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:06:15.821907 24607 webserver.cc:460] Webserver started at http://127.21.182.67:39669/ using document root <none> and password file <none>
I20260810 18:06:15.822510 24607 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:15.822571 24607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:15.822861 24607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:15.825117 24607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "27ce3ca724224c498ef32a6e8e1c623b"
format_stamp: "Formatted at 2026-08-10 18:06:15 on dist-test-slave-27sr"
server_key: "38e2db941600285c9b704b79791ef68d"
server_key_iv: "2e8893b9b65d1d713f35a37c403a013b"
server_key_version: "encryptionkey@0"
I20260810 18:06:15.825881 24607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "27ce3ca724224c498ef32a6e8e1c623b"
format_stamp: "Formatted at 2026-08-10 18:06:15 on dist-test-slave-27sr"
server_key: "38e2db941600285c9b704b79791ef68d"
server_key_iv: "2e8893b9b65d1d713f35a37c403a013b"
server_key_version: "encryptionkey@0"
I20260810 18:06:15.830098 24607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:06:15.832926 24623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:15.834322 24607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:06:15.834479 24607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "27ce3ca724224c498ef32a6e8e1c623b"
format_stamp: "Formatted at 2026-08-10 18:06:15 on dist-test-slave-27sr"
server_key: "38e2db941600285c9b704b79791ef68d"
server_key_iv: "2e8893b9b65d1d713f35a37c403a013b"
server_key_version: "encryptionkey@0"
I20260810 18:06:15.834620 24607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:15.845403 24607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:15.848837 24607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:15.849053 24607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:15.850672 24607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:15.850742 24607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:15.850795 24607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:15.850828 24607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:06:15 dist-test-slave-27sr krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385175, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:15.864194 24607 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:34467
I20260810 18:06:15.864277 24748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:34467 every 8 connection(s)
I20260810 18:06:15.865434 24607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:06:15.866848 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.851724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:33925 (local address 127.21.182.126:46085)
0810 18:06:15.851959 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:15.851965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:15.853059 (+ 1094us) server_negotiation.cc:407] Connection header received
0810 18:06:15.854322 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:15.854326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:15.854389 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:15.854497 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:15.856194 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.856798 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.858698 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.859078 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.859204 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.862499 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:15.862503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:15.862506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:15.862544 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:15.864307 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.864946 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.864949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.864950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.865032 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.865436 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.865439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.865440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.865686 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:15.865806 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:15.866583 (+ 777us) server_negotiation.cc:299] Negotiation successful
0810 18:06:15.866710 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:06:15.867650 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.852357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:46085 (local address 127.21.182.67:33925)
0810 18:06:15.852897 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:15.852943 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:15.853990 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:15.854695 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:15.854713 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:15.855245 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:15.856027 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.856053 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.856969 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.856976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.858499 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.858511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.859254 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.859259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.859495 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.860191 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:15.860208 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:15.862300 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:15.864451 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:15.864458 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:15.864473 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:15.864817 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:15.865131 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:15.865134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:15.865136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:15.865316 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:15.865823 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:15.865841 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:15.866196 (+ 355us) client_negotiation.cc:769] Sending connection context
0810 18:06:15.866466 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 18:06:15.866749 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":147,"threads_started":1}
I20260810 18:06:15.868957 24749 heartbeater.cc:347] Connected to a master server at 127.21.182.126:46085
I20260810 18:06:15.869107 24749 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:15.869696 24749 heartbeater.cc:511] Master 127.21.182.126:46085 requested a full tablet report, sending...
I20260810 18:06:15.871165 24217 ts_manager.cc:194] Registered new tserver with Master: 27ce3ca724224c498ef32a6e8e1c623b (127.21.182.67:34467)
I20260810 18:06:15.871266 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 24607
I20260810 18:06:15.871379 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:06:15.871694 22233 external_mini_cluster.cc:1428] Setting key 12c8f1be3c2a0276b15a61535334dca7
I20260810 18:06:15.872043 24217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:33925
I20260810 18:06:15.874040 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:06:15.880820 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.871505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:46085 (local address 127.21.182.67:43463)
0810 18:06:15.871874 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:15.871931 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:15.872263 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:15.872572 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:15.872575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:15.872850 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:15.873001 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.873009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.873778 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.873782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.874648 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.874661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.875143 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.875149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.875283 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.875933 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:15.875964 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:15.876642 (+ 678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:15.879108 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:15.879117 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:15.879118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:15.879346 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:15.879678 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:15.879680 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:15.879682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:15.879826 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:15.880348 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:15.880351 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:15.880442 (+ 91us) client_negotiation.cc:769] Sending connection context
0810 18:06:15.880528 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 18:06:15.880652 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":118,"threads_started":1}
I20260810 18:06:15.881057 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.871560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:43463 (local address 127.21.182.126:46085)
0810 18:06:15.871688 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:15.871693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:15.872177 (+ 484us) server_negotiation.cc:407] Connection header received
0810 18:06:15.872390 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:15.872393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:15.872452 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:15.872554 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:15.873127 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.873645 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.874793 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.875010 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.875097 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.876811 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:15.876814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:15.876816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:15.876841 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:15.878966 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.879456 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.879462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.879467 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.879570 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.879957 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.879963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.879967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.880246 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:15.880358 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:15.880638 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 18:06:15.880820 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 18:06:15 dist-test-slave-27sr krb5kdc[24167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:06:15 dist-test-slave-27sr krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:15.906199 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.897571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.21.182.126:46085)
0810 18:06:15.897708 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:15.897712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:15.897723 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:06:15.897774 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:15.897778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:15.897821 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:15.897918 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:15.898688 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.899228 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.900184 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.900414 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.900529 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.903019 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:15.903023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:15.903025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:15.903051 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:15.904438 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.904888 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.904891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.904893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.904967 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:15.905253 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:15.905256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:15.905258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:15.905471 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:15.905610 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:15.905869 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 18:06:15.905990 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:06:15.909403 24217 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:06:15.912516 24217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:15.927682 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.921396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.21.182.66:39377)
0810 18:06:15.922109 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:15.922135 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:15.922591 (+ 456us) server_negotiation.cc:407] Connection header received
0810 18:06:15.922685 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:15.922690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:15.922933 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:15.923133 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:15.923427 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.924584 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.926222 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.926941 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.927074 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.927310 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:15.927412 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 18:06:15.927510 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":124,"threads_started":1}
I20260810 18:06:15.927671 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.921200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:39377 (local address 127.0.0.1:45968)
0810 18:06:15.922188 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:15.922270 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:15.922412 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:15.923111 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:15.923117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:15.923140 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:15.923294 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.923304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.924703 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.924706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.926074 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.926086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.927074 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.927078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.927197 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.927249 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 18:06:15.927381 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 18:06:15.927516 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":88,"thread_start_us":70,"threads_started":1}
I20260810 18:06:15.928033 24771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.921690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.21.182.65:45337)
0810 18:06:15.922021 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:15.922030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:15.922373 (+ 343us) server_negotiation.cc:407] Connection header received
0810 18:06:15.922615 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:15.922625 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:15.922918 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:15.923049 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:15.923457 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.924387 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.926224 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.926968 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.927074 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.927231 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:15.927742 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 18:06:15.927857 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
I20260810 18:06:15.928282 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.921515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:45337 (local address 127.0.0.1:43502)
0810 18:06:15.922187 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:15.922251 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:15.922449 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:15.923229 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:15.923231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:15.923255 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:15.923332 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.923341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.924498 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.924502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.926080 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.926086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.927316 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.927323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.927418 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.927441 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:06:15.927509 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 18:06:15.927594 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":112,"spinlock_wait_cycles":1280,"thread_start_us":92,"threads_started":1}
I20260810 18:06:15.928867 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.921979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34467 (local address 127.0.0.1:56306)
0810 18:06:15.923013 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:15.923030 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:15.923258 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:15.924000 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:15.924003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:15.924022 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:15.924106 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.924111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.925410 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.925413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.926395 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:15.926406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.927598 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.927602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:15.927698 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.927721 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:06:15.928637 (+ 916us) client_negotiation.cc:240] Negotiation successful
0810 18:06:15.928729 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":9,"spinlock_wait_cycles":1920,"thread_start_us":95,"threads_started":1}
I20260810 18:06:15.929260 24774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:15.922530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56306 (local address 127.21.182.67:34467)
0810 18:06:15.923537 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:15.923548 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:15.923569 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:06:15.923639 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:15.923643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:15.923823 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:15.923999 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:15.924225 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.925276 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.926531 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:15.927292 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:15.927393 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:15.927541 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:15.929042 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0810 18:06:15.929142 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":121,"threads_started":1}
I20260810 18:06:15.931085 24525 tablet_service.cc:1467] Processing CreateTablet for tablet bb484504eed9453e8e92f5b22d043197 (DEFAULT_TABLE table=test-table [id=adad70a98b1b432c980144759eadf069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:15.931244 24367 tablet_service.cc:1467] Processing CreateTablet for tablet bb484504eed9453e8e92f5b22d043197 (DEFAULT_TABLE table=test-table [id=adad70a98b1b432c980144759eadf069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:15.932328 24525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb484504eed9453e8e92f5b22d043197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:15.931934 24676 tablet_service.cc:1467] Processing CreateTablet for tablet bb484504eed9453e8e92f5b22d043197 (DEFAULT_TABLE table=test-table [id=adad70a98b1b432c980144759eadf069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:15.932332 24367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb484504eed9453e8e92f5b22d043197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:15.932876 24676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb484504eed9453e8e92f5b22d043197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:15.939455 24775 tablet_bootstrap.cc:492] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb: Bootstrap starting.
I20260810 18:06:15.940999 24776 tablet_bootstrap.cc:492] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44: Bootstrap starting.
I20260810 18:06:15.941496 24777 tablet_bootstrap.cc:492] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b: Bootstrap starting.
I20260810 18:06:15.942318 24775 tablet_bootstrap.cc:654] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:15.943286 24775 log.cc:824] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:15.943348 24776 tablet_bootstrap.cc:654] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:15.943441 24777 tablet_bootstrap.cc:654] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:15.944196 24776 log.cc:824] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:15.944311 24777 log.cc:824] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:15.945806 24775 tablet_bootstrap.cc:492] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb: No bootstrap required, opened a new log
I20260810 18:06:15.946048 24775 ts_tablet_manager.cc:1397] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 18:06:15.946342 24776 tablet_bootstrap.cc:492] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44: No bootstrap required, opened a new log
I20260810 18:06:15.946342 24777 tablet_bootstrap.cc:492] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b: No bootstrap required, opened a new log
I20260810 18:06:15.946540 24776 ts_tablet_manager.cc:1397] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:06:15.946540 24777 ts_tablet_manager.cc:1397] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 18:06:15.949867 24775 raft_consensus.cc:348] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:15.950192 24775 raft_consensus.cc:374] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:15.950261 24775 raft_consensus.cc:729] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b3f230abf543aab1f0567731bbd5bb, State: Initialized, Role: FOLLOWER
I20260810 18:06:15.950582 24777 raft_consensus.cc:348] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:15.950582 24776 raft_consensus.cc:348] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:15.950841 24776 raft_consensus.cc:374] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:15.950841 24777 raft_consensus.cc:374] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:15.950802 24775 consensus_queue.cc:260] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:15.950896 24776 raft_consensus.cc:729] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a614cf9a15b343208e90f06c52901d44, State: Initialized, Role: FOLLOWER
I20260810 18:06:15.950906 24777 raft_consensus.cc:729] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ce3ca724224c498ef32a6e8e1c623b, State: Initialized, Role: FOLLOWER
I20260810 18:06:15.951395 24776 consensus_queue.cc:260] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:15.951395 24777 consensus_queue.cc:260] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:15.952088 24439 heartbeater.cc:503] Master 127.21.182.126:46085 was elected leader, sending a full tablet report...
I20260810 18:06:15.952462 24776 ts_tablet_manager.cc:1428] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:06:15.952559 24598 heartbeater.cc:503] Master 127.21.182.126:46085 was elected leader, sending a full tablet report...
I20260810 18:06:15.952075 24775 ts_tablet_manager.cc:1428] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:06:15.952854 24749 heartbeater.cc:503] Master 127.21.182.126:46085 was elected leader, sending a full tablet report...
I20260810 18:06:15.953244 24777 ts_tablet_manager.cc:1428] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260810 18:06:16.116783 24750 tablet.cc:2369] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:06:16.133342 24599 tablet.cc:2369] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:06:16.140376 24441 tablet.cc:2369] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:06:16.204638 24781 raft_consensus.cc:482] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:16.204953 24781 raft_consensus.cc:504] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.206354 24781 leader_election.cc:288] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377), 27ce3ca724224c498ef32a6e8e1c623b (127.21.182.67:34467)
I20260810 18:06:16.210757 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.206928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:52931 (local address 127.21.182.66:39377)
0810 18:06:16.207074 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:16.207081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:16.207281 (+ 200us) server_negotiation.cc:407] Connection header received
0810 18:06:16.207534 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:16.207539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:16.207605 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:16.207693 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:16.208076 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.208709 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.209791 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.210190 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.210324 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.210378 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:16.210557 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 18:06:16.210632 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:06:16.210770 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.206706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:39377 (local address 127.21.182.65:52931)
0810 18:06:16.207132 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:16.207147 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:16.207320 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:16.207716 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:16.207719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:16.207742 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:16.207848 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.207853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.208844 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.208855 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.209644 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.209653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.210319 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.210323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.210421 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.210449 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:06:16.210539 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 18:06:16.210607 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":213,"threads_started":1}
I20260810 18:06:16.211009 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.207325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34467 (local address 127.21.182.65:50073)
0810 18:06:16.207769 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:16.207787 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:16.207950 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:16.208244 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:16.208247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:16.208263 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:16.208388 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.208393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.209127 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.209133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.209901 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.209910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.210551 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.210554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.210661 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.210691 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 18:06:16.210787 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 18:06:16.210850 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":80,"threads_started":1}
I20260810 18:06:16.210995 24774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.207461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:50073 (local address 127.21.182.67:34467)
0810 18:06:16.207588 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:16.207591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:16.207886 (+ 295us) server_negotiation.cc:407] Connection header received
0810 18:06:16.208083 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:16.208089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:16.208143 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:16.208233 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:16.208517 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.209016 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.210029 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.210427 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.210558 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.210612 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:16.210805 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 18:06:16.210874 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:06:16.211840 24696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb484504eed9453e8e92f5b22d043197" candidate_uuid: "41b3f230abf543aab1f0567731bbd5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ce3ca724224c498ef32a6e8e1c623b" is_pre_election: true
I20260810 18:06:16.211845 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb484504eed9453e8e92f5b22d043197" candidate_uuid: "41b3f230abf543aab1f0567731bbd5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a614cf9a15b343208e90f06c52901d44" is_pre_election: true
I20260810 18:06:16.212203 24545 raft_consensus.cc:2454] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41b3f230abf543aab1f0567731bbd5bb in term 0.
I20260810 18:06:16.212218 24696 raft_consensus.cc:2454] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41b3f230abf543aab1f0567731bbd5bb in term 0.
I20260810 18:06:16.212706 24315 leader_election.cc:302] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41b3f230abf543aab1f0567731bbd5bb, a614cf9a15b343208e90f06c52901d44; no voters:
I20260810 18:06:16.213034 24781 raft_consensus.cc:2781] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:16.213143 24781 raft_consensus.cc:482] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:16.213207 24781 raft_consensus.cc:3037] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:16.214493 24781 raft_consensus.cc:504] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.214848 24781 leader_election.cc:288] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [CANDIDATE]: Term 1 election: Requested vote from peers a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377), 27ce3ca724224c498ef32a6e8e1c623b (127.21.182.67:34467)
I20260810 18:06:16.215220 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb484504eed9453e8e92f5b22d043197" candidate_uuid: "41b3f230abf543aab1f0567731bbd5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a614cf9a15b343208e90f06c52901d44"
I20260810 18:06:16.215303 24696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb484504eed9453e8e92f5b22d043197" candidate_uuid: "41b3f230abf543aab1f0567731bbd5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ce3ca724224c498ef32a6e8e1c623b"
I20260810 18:06:16.215348 24545 raft_consensus.cc:3037] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:16.215487 24696 raft_consensus.cc:3037] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:16.216542 24545 raft_consensus.cc:2454] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41b3f230abf543aab1f0567731bbd5bb in term 1.
I20260810 18:06:16.216862 24315 leader_election.cc:302] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41b3f230abf543aab1f0567731bbd5bb, a614cf9a15b343208e90f06c52901d44; no voters:
I20260810 18:06:16.217041 24781 raft_consensus.cc:2781] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:16.217295 24696 raft_consensus.cc:2454] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41b3f230abf543aab1f0567731bbd5bb in term 1.
I20260810 18:06:16.217295 24781 raft_consensus.cc:686] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [term 1 LEADER]: Becoming Leader. State: Replica: 41b3f230abf543aab1f0567731bbd5bb, State: Running, Role: LEADER
I20260810 18:06:16.217662 24781 consensus_queue.cc:237] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.220626 24216 catalog_manager.cc:5495] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 41b3f230abf543aab1f0567731bbd5bb (127.21.182.65). New cstate: current_term: 1 leader_uuid: "41b3f230abf543aab1f0567731bbd5bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } health_report { overall_health: UNKNOWN } } }
Aug 10 18:06:16 dist-test-slave-27sr krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385175, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:16.326781 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.317850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.21.182.126:46085)
0810 18:06:16.318020 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:16.318026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:16.318217 (+ 191us) server_negotiation.cc:407] Connection header received
0810 18:06:16.318406 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:16.318412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:16.318452 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:16.318536 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:16.319257 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.319778 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.320606 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.320821 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.320930 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.323439 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:16.323443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:16.323445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:16.323472 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:16.325007 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:16.325499 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:16.325502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:16.325503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:16.325576 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:16.325855 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:16.325857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:16.325859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:16.326026 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:16.326103 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:16.326501 (+ 398us) server_negotiation.cc:299] Negotiation successful
0810 18:06:16.326634 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":47}
I20260810 18:06:16.326805 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.317776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:46085 (local address 127.0.0.1:40946)
0810 18:06:16.318119 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:16.318132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:16.318300 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:16.318558 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:16.318561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:16.319034 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:16.319132 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.319139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.319918 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.319923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.320483 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.320491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.320946 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.320950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.321033 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.321596 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:16.321608 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:16.323268 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:16.325126 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:16.325136 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:16.325141 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:16.325408 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:16.325675 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:16.325677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:16.325679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:16.325770 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:16.326214 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:16.326220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:16.326342 (+ 122us) client_negotiation.cc:769] Sending connection context
0810 18:06:16.326446 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 18:06:16.326576 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":122,"threads_started":1}
I20260810 18:06:16.331089 24216 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:40946:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 18:06:16.331863 24216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:16.337419 24525 tablet_service.cc:1467] Processing CreateTablet for tablet cec1c007253e4f01a3ecccffe2947076 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08c825f3e14744119281cf84bdd1adce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:06:16.337571 24676 tablet_service.cc:1467] Processing CreateTablet for tablet cec1c007253e4f01a3ecccffe2947076 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08c825f3e14744119281cf84bdd1adce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:06:16.337390 24367 tablet_service.cc:1467] Processing CreateTablet for tablet cec1c007253e4f01a3ecccffe2947076 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08c825f3e14744119281cf84bdd1adce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:06:16.338261 24676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec1c007253e4f01a3ecccffe2947076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:16.338261 24525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec1c007253e4f01a3ecccffe2947076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:16.338260 24367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec1c007253e4f01a3ecccffe2947076. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:16.341408 24777 tablet_bootstrap.cc:492] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b: Bootstrap starting.
I20260810 18:06:16.341475 24775 tablet_bootstrap.cc:492] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb: Bootstrap starting.
I20260810 18:06:16.342379 24777 tablet_bootstrap.cc:654] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:16.342381 24775 tablet_bootstrap.cc:654] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:16.343506 24776 tablet_bootstrap.cc:492] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44: Bootstrap starting.
I20260810 18:06:16.343811 24777 tablet_bootstrap.cc:492] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b: No bootstrap required, opened a new log
I20260810 18:06:16.343950 24777 ts_tablet_manager.cc:1397] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:16.344163 24775 tablet_bootstrap.cc:492] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb: No bootstrap required, opened a new log
I20260810 18:06:16.344267 24775 ts_tablet_manager.cc:1397] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:16.344460 24777 raft_consensus.cc:348] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.344580 24777 raft_consensus.cc:374] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:16.344619 24777 raft_consensus.cc:729] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ce3ca724224c498ef32a6e8e1c623b, State: Initialized, Role: FOLLOWER
I20260810 18:06:16.344655 24776 tablet_bootstrap.cc:654] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:16.344784 24777 consensus_queue.cc:260] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.344753 24775 raft_consensus.cc:348] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.344890 24775 raft_consensus.cc:374] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:16.344930 24775 raft_consensus.cc:729] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b3f230abf543aab1f0567731bbd5bb, State: Initialized, Role: FOLLOWER
I20260810 18:06:16.345075 24775 consensus_queue.cc:260] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.345083 24782 tablet_replica.cc:439] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } } }
I20260810 18:06:16.345258 24782 tablet_replica.cc:442] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:16.345350 24775 ts_tablet_manager.cc:1428] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:06:16.345517 24777 ts_tablet_manager.cc:1428] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:06:16.345829 24781 tablet_replica.cc:439] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } } }
I20260810 18:06:16.345960 24781 tablet_replica.cc:442] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:16.347101 24776 tablet_bootstrap.cc:492] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44: No bootstrap required, opened a new log
I20260810 18:06:16.347199 24776 ts_tablet_manager.cc:1397] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 18:06:16.347721 24776 raft_consensus.cc:348] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.347851 24776 raft_consensus.cc:374] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:16.347910 24776 raft_consensus.cc:729] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a614cf9a15b343208e90f06c52901d44, State: Initialized, Role: FOLLOWER
I20260810 18:06:16.348062 24776 consensus_queue.cc:260] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.348366 24783 tablet_replica.cc:439] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } } }
I20260810 18:06:16.348528 24783 tablet_replica.cc:442] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:16.348372 24776 ts_tablet_manager.cc:1428] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:06:16.575799 24782 raft_consensus.cc:482] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:16.576066 24782 raft_consensus.cc:504] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.577833 24782 leader_election.cc:288] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41b3f230abf543aab1f0567731bbd5bb (127.21.182.65:45337), a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377)
I20260810 18:06:16.582108 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.578250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:47285 (local address 127.21.182.65:45337)
0810 18:06:16.578553 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:16.578561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:16.578628 (+ 67us) server_negotiation.cc:407] Connection header received
0810 18:06:16.578913 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:16.578916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:16.578961 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:16.579061 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:16.579335 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.579871 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.581025 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.581458 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.581602 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.581668 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:16.581872 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 18:06:16.581958 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":96,"threads_started":1}
I20260810 18:06:16.582118 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.578150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:45337 (local address 127.21.182.67:47285)
0810 18:06:16.578549 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:16.578566 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:16.578708 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:16.579070 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:16.579073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:16.579091 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:16.579222 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.579229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.580047 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.580053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.580888 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.580898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.581594 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.581598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.581709 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.581742 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 18:06:16.581840 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 18:06:16.581929 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":98,"threads_started":1}
I20260810 18:06:16.582266 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.578529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:57107 (local address 127.21.182.66:39377)
0810 18:06:16.578688 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:16.578692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:16.578975 (+ 283us) server_negotiation.cc:407] Connection header received
0810 18:06:16.579134 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:16.579145 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:16.579201 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:16.579286 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:16.579555 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.580135 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.581185 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.581596 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.581741 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.581797 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:16.581997 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 18:06:16.582078 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:06:16.582327 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.578160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:39377 (local address 127.21.182.67:57107)
0810 18:06:16.578870 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:16.578882 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:16.579019 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:16.579310 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:16.579313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:16.579329 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:16.579443 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.579448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.580260 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.580264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.581035 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.581045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.581736 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.581740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.581852 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.581875 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:06:16.581978 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 18:06:16.582056 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":25,"thread_start_us":88,"threads_started":1}
I20260810 18:06:16.583040 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec1c007253e4f01a3ecccffe2947076" candidate_uuid: "27ce3ca724224c498ef32a6e8e1c623b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a614cf9a15b343208e90f06c52901d44" is_pre_election: true
I20260810 18:06:16.583066 24390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec1c007253e4f01a3ecccffe2947076" candidate_uuid: "27ce3ca724224c498ef32a6e8e1c623b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41b3f230abf543aab1f0567731bbd5bb" is_pre_election: true
I20260810 18:06:16.583254 24545 raft_consensus.cc:2454] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27ce3ca724224c498ef32a6e8e1c623b in term 0.
I20260810 18:06:16.583380 24390 raft_consensus.cc:2454] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27ce3ca724224c498ef32a6e8e1c623b in term 0.
I20260810 18:06:16.583823 24624 leader_election.cc:302] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ce3ca724224c498ef32a6e8e1c623b, a614cf9a15b343208e90f06c52901d44; no voters:
I20260810 18:06:16.584324 24782 raft_consensus.cc:2781] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:16.584414 24782 raft_consensus.cc:482] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:16.584443 24782 raft_consensus.cc:3037] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:16.585592 24782 raft_consensus.cc:504] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.586020 24782 leader_election.cc:288] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [CANDIDATE]: Term 1 election: Requested vote from peers 41b3f230abf543aab1f0567731bbd5bb (127.21.182.65:45337), a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377)
I20260810 18:06:16.586534 24390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec1c007253e4f01a3ecccffe2947076" candidate_uuid: "27ce3ca724224c498ef32a6e8e1c623b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41b3f230abf543aab1f0567731bbd5bb"
I20260810 18:06:16.586535 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec1c007253e4f01a3ecccffe2947076" candidate_uuid: "27ce3ca724224c498ef32a6e8e1c623b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a614cf9a15b343208e90f06c52901d44"
I20260810 18:06:16.586683 24390 raft_consensus.cc:3037] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:16.586683 24545 raft_consensus.cc:3037] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:16.587975 24545 raft_consensus.cc:2454] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ce3ca724224c498ef32a6e8e1c623b in term 1.
I20260810 18:06:16.588007 24390 raft_consensus.cc:2454] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ce3ca724224c498ef32a6e8e1c623b in term 1.
I20260810 18:06:16.588368 24625 leader_election.cc:302] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 27ce3ca724224c498ef32a6e8e1c623b, 41b3f230abf543aab1f0567731bbd5bb, a614cf9a15b343208e90f06c52901d44; no voters:
I20260810 18:06:16.588542 24782 raft_consensus.cc:2781] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:16.588723 24782 raft_consensus.cc:686] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [term 1 LEADER]: Becoming Leader. State: Replica: 27ce3ca724224c498ef32a6e8e1c623b, State: Running, Role: LEADER
I20260810 18:06:16.589134 24782 consensus_queue.cc:237] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.591687 24805 tablet_replica.cc:439] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b: TxnStatusTablet state changed. Reason: New leader 27ce3ca724224c498ef32a6e8e1c623b. Latest consensus state: current_term: 1 leader_uuid: "27ce3ca724224c498ef32a6e8e1c623b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } } }
I20260810 18:06:16.591825 24805 tablet_replica.cc:442] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:06:16.592164 24807 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:06:16.592418 24807 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 18:06:16.593140 24213 catalog_manager.cc:5495] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b reported cstate change: term changed from 0 to 1, leader changed from <none> to 27ce3ca724224c498ef32a6e8e1c623b (127.21.182.67). New cstate: current_term: 1 leader_uuid: "27ce3ca724224c498ef32a6e8e1c623b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } health_report { overall_health: HEALTHY } } }
I20260810 18:06:16.646467 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.641618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34467 (local address 127.0.0.1:56310)
0810 18:06:16.641986 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:16.642002 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:16.642118 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:16.642493 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:16.642496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:16.642505 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:06:16.642628 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.642635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.643810 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.643813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.644527 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.644538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.645108 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.645112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.645191 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.645257 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:06:16.646126 (+ 869us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:06:16.646146 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:06:16.646231 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 18:06:16.646331 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
I20260810 18:06:16.646476 24774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.641946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.21.182.67:34467)
0810 18:06:16.642081 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:16.642086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:16.642103 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:06:16.642264 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:16.642268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:16.642337 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:16.642457 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:16.642757 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.643552 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.644656 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.644958 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.645076 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.645418 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:16.645992 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:16.646112 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:16.646264 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 18:06:16.646333 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:06:16.653308 24545 raft_consensus.cc:1261] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Refusing update from remote peer 27ce3ca724224c498ef32a6e8e1c623b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:16.653308 24390 raft_consensus.cc:1261] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb [term 1 FOLLOWER]: Refusing update from remote peer 27ce3ca724224c498ef32a6e8e1c623b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:16.654083 24805 consensus_queue.cc:1035] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:16.654335 24782 consensus_queue.cc:1035] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b [LEADER]: Connected to new peer: Peer: permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:16.657770 24781 tablet_replica.cc:439] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb: TxnStatusTablet state changed. Reason: New leader 27ce3ca724224c498ef32a6e8e1c623b. Latest consensus state: current_term: 1 leader_uuid: "27ce3ca724224c498ef32a6e8e1c623b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } } }
I20260810 18:06:16.657915 24781 tablet_replica.cc:442] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:16.659198 24783 tablet_replica.cc:439] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44: TxnStatusTablet state changed. Reason: New leader 27ce3ca724224c498ef32a6e8e1c623b. Latest consensus state: current_term: 1 leader_uuid: "27ce3ca724224c498ef32a6e8e1c623b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } } }
I20260810 18:06:16.659360 24783 tablet_replica.cc:442] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:16.662928 24782 tablet_replica.cc:439] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27ce3ca724224c498ef32a6e8e1c623b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } } }
I20260810 18:06:16.663070 24782 tablet_replica.cc:442] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:06:16.663259 24813 mvcc.cc:205] Tried to move back new op lower bound from 7317033683562991616 to 7317033683312242688. Current Snapshot: MvccSnapshot[applied={T|T < 7317033683562991616}]
I20260810 18:06:16.663424 24782 tablet_replica.cc:439] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27ce3ca724224c498ef32a6e8e1c623b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } } }
I20260810 18:06:16.663525 24782 tablet_replica.cc:442] T cec1c007253e4f01a3ecccffe2947076 P 27ce3ca724224c498ef32a6e8e1c623b: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:06:16.664784 24783 tablet_replica.cc:439] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27ce3ca724224c498ef32a6e8e1c623b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } } }
I20260810 18:06:16.664911 24783 tablet_replica.cc:442] T cec1c007253e4f01a3ecccffe2947076 P a614cf9a15b343208e90f06c52901d44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:16.665822 24814 mvcc.cc:205] Tried to move back new op lower bound from 7317033683562991616 to 7317033683312242688. Current Snapshot: MvccSnapshot[applied={T|T < 7317033683562991616}]
I20260810 18:06:16.666743 24781 tablet_replica.cc:439] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27ce3ca724224c498ef32a6e8e1c623b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } } }
I20260810 18:06:16.666872 24781 tablet_replica.cc:442] T cec1c007253e4f01a3ecccffe2947076 P 41b3f230abf543aab1f0567731bbd5bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:16.667039 24815 mvcc.cc:205] Tried to move back new op lower bound from 7317033683562991616 to 7317033683312242688. Current Snapshot: MvccSnapshot[applied={T|T < 7317033683562991616}]
I20260810 18:06:16.675681 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.671124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.21.182.65:45337)
0810 18:06:16.671248 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:16.671253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:16.671296 (+ 43us) server_negotiation.cc:407] Connection header received
0810 18:06:16.671374 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:16.671377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:16.671442 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:16.671554 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:16.671939 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.672550 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.673919 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.674158 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.674265 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.674648 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:16.675213 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:16.675324 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:16.675435 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 18:06:16.675496 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:06:16.689896 24774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.685507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:40369 (local address 127.21.182.67:34467)
0810 18:06:16.685678 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:16.685684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:16.685874 (+ 190us) server_negotiation.cc:407] Connection header received
0810 18:06:16.686024 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:16.686027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:16.686077 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:16.686152 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:16.686496 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.687251 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.688689 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.688927 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.689055 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.689428 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:16.689511 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:16.689577 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:16.689699 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 18:06:16.689760 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:06:16.689947 24827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.685404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34467 (local address 127.21.182.65:40369)
0810 18:06:16.685769 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:16.685785 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:16.685913 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:16.686214 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:16.686217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:16.686227 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:06:16.686378 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.686384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.687402 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.687406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.688540 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.688553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.689054 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.689057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.689157 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.689229 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:06:16.689605 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:06:16.689612 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:06:16.689676 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 18:06:16.689770 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":140,"threads_started":1}
I20260810 18:06:16.698675 24545 raft_consensus.cc:1261] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Refusing update from remote peer 41b3f230abf543aab1f0567731bbd5bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:16.698894 24696 raft_consensus.cc:1261] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Refusing update from remote peer 41b3f230abf543aab1f0567731bbd5bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:16.699419 24787 consensus_queue.cc:1035] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:16.699868 24787 consensus_queue.cc:1035] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:16.731220 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.726939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:47253 (local address 127.21.182.65:45337)
0810 18:06:16.727085 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:16.727091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:16.727421 (+ 330us) server_negotiation.cc:407] Connection header received
0810 18:06:16.727606 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:16.727609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:16.727677 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:16.727768 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:16.728159 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.728797 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.729911 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.730168 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.730301 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.730664 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:16.730787 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:16.730878 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:16.730985 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 18:06:16.731044 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:06:16.731266 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:16.726886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:45337 (local address 127.21.182.67:47253)
0810 18:06:16.727307 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:16.727349 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:16.727473 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:16.727793 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:16.727796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:16.727805 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:06:16.728024 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.728031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.728944 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.728950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.729770 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:16.729781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:16.730317 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:16.730321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:16.730433 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:16.730487 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:06:16.730883 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:06:16.730900 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 18:06:16.730973 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 18:06:16.731063 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":133,"threads_started":1}
I20260810 18:06:16.769595 24217 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:16.769856 24217 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:16.773173 24217 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf: Sending DeleteTablet for 3 replicas of tablet bb484504eed9453e8e92f5b22d043197
I20260810 18:06:16.774441 24367 tablet_service.cc:1514] Processing DeleteTablet for tablet bb484504eed9453e8e92f5b22d043197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:16 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:43502
I20260810 18:06:16.774528 24525 tablet_service.cc:1514] Processing DeleteTablet for tablet bb484504eed9453e8e92f5b22d043197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:16 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:45968
I20260810 18:06:16.775069 24676 tablet_service.cc:1514] Processing DeleteTablet for tablet bb484504eed9453e8e92f5b22d043197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:16 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:56306
I20260810 18:06:16.775324 24836 tablet_replica.cc:330] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb: stopping tablet replica
I20260810 18:06:16.775732 24836 raft_consensus.cc:2229] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:06:16.776394 24839 tablet_replica.cc:330] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b: stopping tablet replica
I20260810 18:06:16.776772 24839 raft_consensus.cc:2229] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:16.776679 24217 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 18:06:16.777143 24839 raft_consensus.cc:2258] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 18:06:16.777141 24217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:16.777422 24837 tablet_replica.cc:330] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44: stopping tablet replica
I20260810 18:06:16.777966 24836 raft_consensus.cc:2258] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:16.779524 24836 ts_tablet_manager.cc:1905] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:16.783063 24367 tablet_service.cc:1467] Processing CreateTablet for tablet eb8ea98434044534b418058b80eb3fcd (DEFAULT_TABLE table=test-table [id=786646b0e14f467cba7c56c19d7e388d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:16.783357 24367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8ea98434044534b418058b80eb3fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:16.784790 24836 ts_tablet_manager.cc:1918] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:06:16.784909 24836 log.cc:1196] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/wal/wals/bb484504eed9453e8e92f5b22d043197
I20260810 18:06:16.785158 24525 tablet_service.cc:1467] Processing CreateTablet for tablet eb8ea98434044534b418058b80eb3fcd (DEFAULT_TABLE table=test-table [id=786646b0e14f467cba7c56c19d7e388d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:16.785274 24836 ts_tablet_manager.cc:1939] T bb484504eed9453e8e92f5b22d043197 P 41b3f230abf543aab1f0567731bbd5bb: Deleting consensus metadata
I20260810 18:06:16.785483 24525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8ea98434044534b418058b80eb3fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:16.787070 24200 catalog_manager.cc:4841] TS 41b3f230abf543aab1f0567731bbd5bb (127.21.182.65:45337): tablet bb484504eed9453e8e92f5b22d043197 (table test-table [id=adad70a98b1b432c980144759eadf069]) successfully deleted
I20260810 18:06:16.788051 24776 tablet_bootstrap.cc:492] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44: Bootstrap starting.
I20260810 18:06:16.788194 24676 tablet_service.cc:1467] Processing CreateTablet for tablet eb8ea98434044534b418058b80eb3fcd (DEFAULT_TABLE table=test-table [id=786646b0e14f467cba7c56c19d7e388d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:16.788523 24676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8ea98434044534b418058b80eb3fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:16.789152 24776 tablet_bootstrap.cc:654] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:16.790558 24776 tablet_bootstrap.cc:492] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44: No bootstrap required, opened a new log
I20260810 18:06:16.790647 24776 ts_tablet_manager.cc:1397] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 18:06:16.791055 24777 tablet_bootstrap.cc:492] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b: Bootstrap starting.
I20260810 18:06:16.791149 24776 raft_consensus.cc:348] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.791289 24776 raft_consensus.cc:374] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:16.791323 24776 raft_consensus.cc:729] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a614cf9a15b343208e90f06c52901d44, State: Initialized, Role: FOLLOWER
I20260810 18:06:16.791465 24776 consensus_queue.cc:260] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.791756 24776 ts_tablet_manager.cc:1428] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260810 18:06:16.791921 24777 tablet_bootstrap.cc:654] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:16.795573 24775 tablet_bootstrap.cc:492] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb: Bootstrap starting.
I20260810 18:06:16.796635 24775 tablet_bootstrap.cc:654] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:16.798264 24775 tablet_bootstrap.cc:492] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb: No bootstrap required, opened a new log
I20260810 18:06:16.798365 24775 ts_tablet_manager.cc:1397] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:16.798862 24775 raft_consensus.cc:348] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.798988 24775 raft_consensus.cc:374] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:16.799027 24775 raft_consensus.cc:729] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b3f230abf543aab1f0567731bbd5bb, State: Initialized, Role: FOLLOWER
I20260810 18:06:16.799180 24775 consensus_queue.cc:260] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.799882 24775 ts_tablet_manager.cc:1428] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:06:16.800220 24837 raft_consensus.cc:2229] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:16.800491 24837 raft_consensus.cc:2258] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:16.801749 24839 ts_tablet_manager.cc:1905] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:16.804901 24777 tablet_bootstrap.cc:492] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b: No bootstrap required, opened a new log
I20260810 18:06:16.805024 24777 ts_tablet_manager.cc:1397] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20260810 18:06:16.805193 24839 ts_tablet_manager.cc:1918] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:06:16.805313 24839 log.cc:1196] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/bb484504eed9453e8e92f5b22d043197
I20260810 18:06:16.805545 24777 raft_consensus.cc:348] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.805648 24839 ts_tablet_manager.cc:1939] T bb484504eed9453e8e92f5b22d043197 P 27ce3ca724224c498ef32a6e8e1c623b: Deleting consensus metadata
I20260810 18:06:16.805781 24777 raft_consensus.cc:374] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:16.805830 24777 raft_consensus.cc:729] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ce3ca724224c498ef32a6e8e1c623b, State: Initialized, Role: FOLLOWER
I20260810 18:06:16.805989 24777 consensus_queue.cc:260] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:16.806284 24777 ts_tablet_manager.cc:1428] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:06:16.806469 24837 ts_tablet_manager.cc:1905] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:16.807859 24198 catalog_manager.cc:4841] TS 27ce3ca724224c498ef32a6e8e1c623b (127.21.182.67:34467): tablet bb484504eed9453e8e92f5b22d043197 (table test-table [id=adad70a98b1b432c980144759eadf069]) successfully deleted
I20260810 18:06:16.809738 24837 ts_tablet_manager.cc:1918] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:06:16.809844 24837 log.cc:1196] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/bb484504eed9453e8e92f5b22d043197
I20260810 18:06:16.810531 24837 ts_tablet_manager.cc:1939] T bb484504eed9453e8e92f5b22d043197 P a614cf9a15b343208e90f06c52901d44: Deleting consensus metadata
I20260810 18:06:16.812058 24198 catalog_manager.cc:4841] TS a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377): tablet bb484504eed9453e8e92f5b22d043197 (table test-table [id=adad70a98b1b432c980144759eadf069]) successfully deleted
I20260810 18:06:17.056396 24787 raft_consensus.cc:482] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:17.056596 24787 raft_consensus.cc:504] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:17.057250 24787 leader_election.cc:288] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377), 27ce3ca724224c498ef32a6e8e1c623b (127.21.182.67:34467)
I20260810 18:06:17.057991 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8ea98434044534b418058b80eb3fcd" candidate_uuid: "41b3f230abf543aab1f0567731bbd5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a614cf9a15b343208e90f06c52901d44" is_pre_election: true
I20260810 18:06:17.058035 24696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8ea98434044534b418058b80eb3fcd" candidate_uuid: "41b3f230abf543aab1f0567731bbd5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ce3ca724224c498ef32a6e8e1c623b" is_pre_election: true
I20260810 18:06:17.058218 24696 raft_consensus.cc:2454] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41b3f230abf543aab1f0567731bbd5bb in term 0.
I20260810 18:06:17.058205 24545 raft_consensus.cc:2454] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41b3f230abf543aab1f0567731bbd5bb in term 0.
I20260810 18:06:17.058629 24315 leader_election.cc:302] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ce3ca724224c498ef32a6e8e1c623b, 41b3f230abf543aab1f0567731bbd5bb; no voters:
I20260810 18:06:17.058990 24787 raft_consensus.cc:2781] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:17.059055 24787 raft_consensus.cc:482] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:17.059083 24787 raft_consensus.cc:3037] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:17.060279 24787 raft_consensus.cc:504] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:17.060688 24787 leader_election.cc:288] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [CANDIDATE]: Term 1 election: Requested vote from peers a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377), 27ce3ca724224c498ef32a6e8e1c623b (127.21.182.67:34467)
I20260810 18:06:17.061245 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8ea98434044534b418058b80eb3fcd" candidate_uuid: "41b3f230abf543aab1f0567731bbd5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a614cf9a15b343208e90f06c52901d44"
I20260810 18:06:17.061328 24696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8ea98434044534b418058b80eb3fcd" candidate_uuid: "41b3f230abf543aab1f0567731bbd5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ce3ca724224c498ef32a6e8e1c623b"
I20260810 18:06:17.061422 24545 raft_consensus.cc:3037] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:17.061515 24696 raft_consensus.cc:3037] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:17.062572 24696 raft_consensus.cc:2454] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41b3f230abf543aab1f0567731bbd5bb in term 1.
I20260810 18:06:17.062665 24545 raft_consensus.cc:2454] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41b3f230abf543aab1f0567731bbd5bb in term 1.
I20260810 18:06:17.062943 24315 leader_election.cc:302] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ce3ca724224c498ef32a6e8e1c623b, 41b3f230abf543aab1f0567731bbd5bb; no voters:
I20260810 18:06:17.063184 24787 raft_consensus.cc:2781] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:17.063308 24787 raft_consensus.cc:686] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [term 1 LEADER]: Becoming Leader. State: Replica: 41b3f230abf543aab1f0567731bbd5bb, State: Running, Role: LEADER
I20260810 18:06:17.063486 24787 consensus_queue.cc:237] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:17.065797 24213 catalog_manager.cc:5495] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 41b3f230abf543aab1f0567731bbd5bb (127.21.182.65). New cstate: current_term: 1 leader_uuid: "41b3f230abf543aab1f0567731bbd5bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } health_report { overall_health: UNKNOWN } } }
I20260810 18:06:17.107515 24696 raft_consensus.cc:1261] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Refusing update from remote peer 41b3f230abf543aab1f0567731bbd5bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:17.107653 24544 raft_consensus.cc:1261] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Refusing update from remote peer 41b3f230abf543aab1f0567731bbd5bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:17.108027 24781 consensus_queue.cc:1035] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:17.108193 24787 consensus_queue.cc:1035] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:06:17.123344 24815 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 18:06:18.678411 24213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:18.678588 24213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:18.681447 24213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf: Sending DeleteTablet for 3 replicas of tablet eb8ea98434044534b418058b80eb3fcd
I20260810 18:06:18.682251 24367 tablet_service.cc:1514] Processing DeleteTablet for tablet eb8ea98434044534b418058b80eb3fcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:18 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:43502
I20260810 18:06:18.682432 24676 tablet_service.cc:1514] Processing DeleteTablet for tablet eb8ea98434044534b418058b80eb3fcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:18 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:56306
I20260810 18:06:18.682683 24874 tablet_replica.cc:330] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb: stopping tablet replica
I20260810 18:06:18.682684 24525 tablet_service.cc:1514] Processing DeleteTablet for tablet eb8ea98434044534b418058b80eb3fcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:18 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:45968
I20260810 18:06:18.682822 24874 raft_consensus.cc:2229] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:06:18.682910 24879 tablet_replica.cc:330] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b: stopping tablet replica
I20260810 18:06:18.683096 24879 raft_consensus.cc:2229] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:18.683139 24874 raft_consensus.cc:2258] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:18.683247 24879 raft_consensus.cc:2258] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:18.683475 24880 tablet_replica.cc:330] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44: stopping tablet replica
I20260810 18:06:18.683677 24880 raft_consensus.cc:2229] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:18.683843 24880 raft_consensus.cc:2258] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:18.684291 24874 ts_tablet_manager.cc:1905] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:18.684610 24880 ts_tablet_manager.cc:1905] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:18.687299 24874 ts_tablet_manager.cc:1918] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:06:18.687299 24880 ts_tablet_manager.cc:1918] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:06:18.687407 24874 log.cc:1196] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/wal/wals/eb8ea98434044534b418058b80eb3fcd
I20260810 18:06:18.687439 24880 log.cc:1196] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/eb8ea98434044534b418058b80eb3fcd
I20260810 18:06:18.687753 24880 ts_tablet_manager.cc:1939] T eb8ea98434044534b418058b80eb3fcd P a614cf9a15b343208e90f06c52901d44: Deleting consensus metadata
I20260810 18:06:18.687753 24874 ts_tablet_manager.cc:1939] T eb8ea98434044534b418058b80eb3fcd P 41b3f230abf543aab1f0567731bbd5bb: Deleting consensus metadata
I20260810 18:06:18.688766 24200 catalog_manager.cc:4841] TS 41b3f230abf543aab1f0567731bbd5bb (127.21.182.65:45337): tablet eb8ea98434044534b418058b80eb3fcd (table test-table [id=786646b0e14f467cba7c56c19d7e388d]) successfully deleted
I20260810 18:06:18.688773 24198 catalog_manager.cc:4841] TS a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377): tablet eb8ea98434044534b418058b80eb3fcd (table test-table [id=786646b0e14f467cba7c56c19d7e388d]) successfully deleted
I20260810 18:06:18.689218 24879 ts_tablet_manager.cc:1905] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:18.691794 24879 ts_tablet_manager.cc:1918] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:06:18.691916 24879 log.cc:1196] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/eb8ea98434044534b418058b80eb3fcd
I20260810 18:06:18.692209 24879 ts_tablet_manager.cc:1939] T eb8ea98434044534b418058b80eb3fcd P 27ce3ca724224c498ef32a6e8e1c623b: Deleting consensus metadata
I20260810 18:06:18.693204 24198 catalog_manager.cc:4841] TS 27ce3ca724224c498ef32a6e8e1c623b (127.21.182.67:34467): tablet eb8ea98434044534b418058b80eb3fcd (table test-table [id=786646b0e14f467cba7c56c19d7e388d]) successfully deleted
Aug 10 18:06:18 dist-test-slave-27sr krb5kdc[24167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385178, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:06:18 dist-test-slave-27sr krb5kdc[24167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385178, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:18.716576 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:18.706320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.21.182.126:46085)
0810 18:06:18.706870 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:18.706923 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:18.706939 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:06:18.706987 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:18.706990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:18.707037 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:18.707182 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:18.708272 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:18.708936 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:18.709866 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:18.710105 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:18.710189 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:18.712939 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:18.712942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:18.712944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:18.712974 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:18.714762 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:18.715205 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:18.715207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:18.715209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:18.715293 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:18.715572 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:18.715574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:18.715575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:18.715770 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:18.715980 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:18.716225 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 18:06:18.716343 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":306,"threads_started":1}
I20260810 18:06:18.719414 24213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:06:18.719949 24213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:18.725275 24367 tablet_service.cc:1467] Processing CreateTablet for tablet 508b03fa486e43acb41b4178d7ec064b (DEFAULT_TABLE table=test-table [id=dc62ea3080e24ac7903880098ad15bc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:18.725320 24676 tablet_service.cc:1467] Processing CreateTablet for tablet 508b03fa486e43acb41b4178d7ec064b (DEFAULT_TABLE table=test-table [id=dc62ea3080e24ac7903880098ad15bc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:18.725666 24367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508b03fa486e43acb41b4178d7ec064b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:18.725697 24676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508b03fa486e43acb41b4178d7ec064b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:18.728237 24890 tablet_bootstrap.cc:492] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b: Bootstrap starting.
I20260810 18:06:18.728199 24525 tablet_service.cc:1467] Processing CreateTablet for tablet 508b03fa486e43acb41b4178d7ec064b (DEFAULT_TABLE table=test-table [id=dc62ea3080e24ac7903880098ad15bc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:18.728466 24891 tablet_bootstrap.cc:492] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb: Bootstrap starting.
I20260810 18:06:18.728518 24525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 508b03fa486e43acb41b4178d7ec064b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:18.729288 24891 tablet_bootstrap.cc:654] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:18.729377 24890 tablet_bootstrap.cc:654] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:18.730706 24891 tablet_bootstrap.cc:492] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb: No bootstrap required, opened a new log
I20260810 18:06:18.730818 24891 ts_tablet_manager.cc:1397] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 18:06:18.731410 24891 raft_consensus.cc:348] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.731561 24891 raft_consensus.cc:374] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:18.731597 24891 raft_consensus.cc:729] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b3f230abf543aab1f0567731bbd5bb, State: Initialized, Role: FOLLOWER
I20260810 18:06:18.731755 24894 tablet_bootstrap.cc:492] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44: Bootstrap starting.
I20260810 18:06:18.731730 24891 consensus_queue.cc:260] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.732712 24890 tablet_bootstrap.cc:492] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b: No bootstrap required, opened a new log
I20260810 18:06:18.732879 24890 ts_tablet_manager.cc:1397] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 18:06:18.733345 24894 tablet_bootstrap.cc:654] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:18.733953 24890 raft_consensus.cc:348] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.734138 24891 ts_tablet_manager.cc:1428] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260810 18:06:18.734174 24890 raft_consensus.cc:374] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:18.734249 24890 raft_consensus.cc:729] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ce3ca724224c498ef32a6e8e1c623b, State: Initialized, Role: FOLLOWER
I20260810 18:06:18.734447 24890 consensus_queue.cc:260] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.734742 24890 ts_tablet_manager.cc:1428] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:06:18.735826 24894 tablet_bootstrap.cc:492] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44: No bootstrap required, opened a new log
I20260810 18:06:18.736045 24894 ts_tablet_manager.cc:1397] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 18:06:18.736635 24894 raft_consensus.cc:348] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.736765 24894 raft_consensus.cc:374] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:18.736811 24894 raft_consensus.cc:729] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a614cf9a15b343208e90f06c52901d44, State: Initialized, Role: FOLLOWER
I20260810 18:06:18.736953 24894 consensus_queue.cc:260] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.737448 24894 ts_tablet_manager.cc:1428] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:06:18.767575 24866 raft_consensus.cc:482] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:18.767817 24866 raft_consensus.cc:504] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.768401 24866 leader_election.cc:288] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377), 27ce3ca724224c498ef32a6e8e1c623b (127.21.182.67:34467)
I20260810 18:06:18.769017 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508b03fa486e43acb41b4178d7ec064b" candidate_uuid: "41b3f230abf543aab1f0567731bbd5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a614cf9a15b343208e90f06c52901d44" is_pre_election: true
I20260810 18:06:18.769073 24696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508b03fa486e43acb41b4178d7ec064b" candidate_uuid: "41b3f230abf543aab1f0567731bbd5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ce3ca724224c498ef32a6e8e1c623b" is_pre_election: true
I20260810 18:06:18.769212 24696 raft_consensus.cc:2454] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41b3f230abf543aab1f0567731bbd5bb in term 0.
I20260810 18:06:18.769213 24544 raft_consensus.cc:2454] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41b3f230abf543aab1f0567731bbd5bb in term 0.
I20260810 18:06:18.769583 24315 leader_election.cc:302] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ce3ca724224c498ef32a6e8e1c623b, 41b3f230abf543aab1f0567731bbd5bb; no voters:
I20260810 18:06:18.769770 24866 raft_consensus.cc:2781] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:18.769836 24866 raft_consensus.cc:482] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:18.769892 24866 raft_consensus.cc:3037] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:18.771168 24866 raft_consensus.cc:504] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.771534 24866 leader_election.cc:288] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [CANDIDATE]: Term 1 election: Requested vote from peers a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377), 27ce3ca724224c498ef32a6e8e1c623b (127.21.182.67:34467)
I20260810 18:06:18.772017 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508b03fa486e43acb41b4178d7ec064b" candidate_uuid: "41b3f230abf543aab1f0567731bbd5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a614cf9a15b343208e90f06c52901d44"
I20260810 18:06:18.772065 24696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "508b03fa486e43acb41b4178d7ec064b" candidate_uuid: "41b3f230abf543aab1f0567731bbd5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ce3ca724224c498ef32a6e8e1c623b"
I20260810 18:06:18.772163 24544 raft_consensus.cc:3037] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:18.772182 24696 raft_consensus.cc:3037] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:18.773681 24696 raft_consensus.cc:2454] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41b3f230abf543aab1f0567731bbd5bb in term 1.
I20260810 18:06:18.774058 24544 raft_consensus.cc:2454] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41b3f230abf543aab1f0567731bbd5bb in term 1.
I20260810 18:06:18.774070 24315 leader_election.cc:302] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ce3ca724224c498ef32a6e8e1c623b, 41b3f230abf543aab1f0567731bbd5bb; no voters:
I20260810 18:06:18.774310 24866 raft_consensus.cc:2781] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:18.774408 24866 raft_consensus.cc:686] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [term 1 LEADER]: Becoming Leader. State: Replica: 41b3f230abf543aab1f0567731bbd5bb, State: Running, Role: LEADER
I20260810 18:06:18.774561 24866 consensus_queue.cc:237] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.776598 24213 catalog_manager.cc:5495] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 41b3f230abf543aab1f0567731bbd5bb (127.21.182.65). New cstate: current_term: 1 leader_uuid: "41b3f230abf543aab1f0567731bbd5bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } health_report { overall_health: UNKNOWN } } }
I20260810 18:06:18.803733 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:18.800435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.21.182.65:45337)
0810 18:06:18.800771 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:18.800781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:18.800798 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:06:18.800849 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:18.800851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:18.800895 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:18.800977 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:18.801213 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:18.801768 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:18.802736 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:18.802981 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:18.803073 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:18.803311 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:18.803388 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:18.803443 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:18.803545 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 18:06:18.803600 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":184,"threads_started":1}
I20260810 18:06:18.811635 24696 raft_consensus.cc:1261] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Refusing update from remote peer 41b3f230abf543aab1f0567731bbd5bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:18.811767 24544 raft_consensus.cc:1261] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Refusing update from remote peer 41b3f230abf543aab1f0567731bbd5bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:18.812130 24866 consensus_queue.cc:1035] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:18.812320 24862 consensus_queue.cc:1035] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:18.848737 24216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:18.848950 24216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:18.851135 24216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf: Sending DeleteTablet for 3 replicas of tablet 508b03fa486e43acb41b4178d7ec064b
I20260810 18:06:18.851779 24367 tablet_service.cc:1514] Processing DeleteTablet for tablet 508b03fa486e43acb41b4178d7ec064b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:18 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:43502
I20260810 18:06:18.851924 24676 tablet_service.cc:1514] Processing DeleteTablet for tablet 508b03fa486e43acb41b4178d7ec064b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:18 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:56306
I20260810 18:06:18.852015 24874 tablet_replica.cc:330] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb: stopping tablet replica
I20260810 18:06:18.852093 24525 tablet_service.cc:1514] Processing DeleteTablet for tablet 508b03fa486e43acb41b4178d7ec064b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:18 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:45968
I20260810 18:06:18.852136 24874 raft_consensus.cc:2229] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:06:18.852159 24879 tablet_replica.cc:330] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b: stopping tablet replica
I20260810 18:06:18.852304 24879 raft_consensus.cc:2229] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:18.852373 24880 tablet_replica.cc:330] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44: stopping tablet replica
I20260810 18:06:18.852353 24874 raft_consensus.cc:2258] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:18.852433 24879 raft_consensus.cc:2258] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:18.852505 24880 raft_consensus.cc:2229] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:18.852628 24880 raft_consensus.cc:2258] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:18.853201 24874 ts_tablet_manager.cc:1905] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:18.853263 24879 ts_tablet_manager.cc:1905] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:18.853367 24216 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 18:06:18.853484 24880 ts_tablet_manager.cc:1905] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 18:06:18.853987 24216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:18.856161 24880 ts_tablet_manager.cc:1918] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:06:18.856161 24874 ts_tablet_manager.cc:1918] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:06:18.856251 24880 log.cc:1196] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/508b03fa486e43acb41b4178d7ec064b
I20260810 18:06:18.856251 24874 log.cc:1196] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-0/wal/wals/508b03fa486e43acb41b4178d7ec064b
I20260810 18:06:18.856338 24879 ts_tablet_manager.cc:1918] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:06:18.856417 24879 log.cc:1196] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/508b03fa486e43acb41b4178d7ec064b
I20260810 18:06:18.856546 24874 ts_tablet_manager.cc:1939] T 508b03fa486e43acb41b4178d7ec064b P 41b3f230abf543aab1f0567731bbd5bb: Deleting consensus metadata
I20260810 18:06:18.856554 24880 ts_tablet_manager.cc:1939] T 508b03fa486e43acb41b4178d7ec064b P a614cf9a15b343208e90f06c52901d44: Deleting consensus metadata
I20260810 18:06:18.856694 24879 ts_tablet_manager.cc:1939] T 508b03fa486e43acb41b4178d7ec064b P 27ce3ca724224c498ef32a6e8e1c623b: Deleting consensus metadata
I20260810 18:06:18.857542 24198 catalog_manager.cc:4841] TS 27ce3ca724224c498ef32a6e8e1c623b (127.21.182.67:34467): tablet 508b03fa486e43acb41b4178d7ec064b (table test-table [id=dc62ea3080e24ac7903880098ad15bc2]) successfully deleted
I20260810 18:06:18.857736 24198 catalog_manager.cc:4841] TS a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377): tablet 508b03fa486e43acb41b4178d7ec064b (table test-table [id=dc62ea3080e24ac7903880098ad15bc2]) successfully deleted
I20260810 18:06:18.858176 24200 catalog_manager.cc:4841] TS 41b3f230abf543aab1f0567731bbd5bb (127.21.182.65:45337): tablet 508b03fa486e43acb41b4178d7ec064b (table test-table [id=dc62ea3080e24ac7903880098ad15bc2]) successfully deleted
I20260810 18:06:18.860885 24367 tablet_service.cc:1467] Processing CreateTablet for tablet 459cd050375f48ce9c9fdad4c38b956d (DEFAULT_TABLE table=test-table [id=8224c58a142543758554f08edb293a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:18.861147 24525 tablet_service.cc:1467] Processing CreateTablet for tablet 459cd050375f48ce9c9fdad4c38b956d (DEFAULT_TABLE table=test-table [id=8224c58a142543758554f08edb293a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:18.861217 24367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 459cd050375f48ce9c9fdad4c38b956d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:18.861196 24676 tablet_service.cc:1467] Processing CreateTablet for tablet 459cd050375f48ce9c9fdad4c38b956d (DEFAULT_TABLE table=test-table [id=8224c58a142543758554f08edb293a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:18.861418 24525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 459cd050375f48ce9c9fdad4c38b956d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:18.861547 24676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 459cd050375f48ce9c9fdad4c38b956d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:18.863574 24890 tablet_bootstrap.cc:492] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b: Bootstrap starting.
I20260810 18:06:18.863803 24891 tablet_bootstrap.cc:492] T 459cd050375f48ce9c9fdad4c38b956d P 41b3f230abf543aab1f0567731bbd5bb: Bootstrap starting.
I20260810 18:06:18.863967 24894 tablet_bootstrap.cc:492] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44: Bootstrap starting.
I20260810 18:06:18.864658 24891 tablet_bootstrap.cc:654] T 459cd050375f48ce9c9fdad4c38b956d P 41b3f230abf543aab1f0567731bbd5bb: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:18.864784 24890 tablet_bootstrap.cc:654] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:18.865062 24894 tablet_bootstrap.cc:654] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:18.865875 24891 tablet_bootstrap.cc:492] T 459cd050375f48ce9c9fdad4c38b956d P 41b3f230abf543aab1f0567731bbd5bb: No bootstrap required, opened a new log
I20260810 18:06:18.865967 24891 ts_tablet_manager.cc:1397] T 459cd050375f48ce9c9fdad4c38b956d P 41b3f230abf543aab1f0567731bbd5bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:06:18.866291 24894 tablet_bootstrap.cc:492] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44: No bootstrap required, opened a new log
I20260810 18:06:18.866374 24894 ts_tablet_manager.cc:1397] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 18:06:18.866411 24891 raft_consensus.cc:348] T 459cd050375f48ce9c9fdad4c38b956d P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.866546 24891 raft_consensus.cc:374] T 459cd050375f48ce9c9fdad4c38b956d P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:18.866648 24891 raft_consensus.cc:729] T 459cd050375f48ce9c9fdad4c38b956d P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b3f230abf543aab1f0567731bbd5bb, State: Initialized, Role: FOLLOWER
I20260810 18:06:18.866854 24891 consensus_queue.cc:260] T 459cd050375f48ce9c9fdad4c38b956d P 41b3f230abf543aab1f0567731bbd5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.867137 24891 ts_tablet_manager.cc:1428] T 459cd050375f48ce9c9fdad4c38b956d P 41b3f230abf543aab1f0567731bbd5bb: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 18:06:18.867156 24894 raft_consensus.cc:348] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.867264 24894 raft_consensus.cc:374] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:18.867296 24894 raft_consensus.cc:729] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a614cf9a15b343208e90f06c52901d44, State: Initialized, Role: FOLLOWER
I20260810 18:06:18.867429 24894 consensus_queue.cc:260] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.867692 24894 ts_tablet_manager.cc:1428] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:06:18.868542 24890 tablet_bootstrap.cc:492] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b: No bootstrap required, opened a new log
I20260810 18:06:18.868647 24890 ts_tablet_manager.cc:1397] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 18:06:18.869114 24890 raft_consensus.cc:348] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.869282 24890 raft_consensus.cc:374] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:18.869321 24890 raft_consensus.cc:729] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ce3ca724224c498ef32a6e8e1c623b, State: Initialized, Role: FOLLOWER
I20260810 18:06:18.869453 24890 consensus_queue.cc:260] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.869765 24890 ts_tablet_manager.cc:1428] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:06:18.875962 24847 raft_consensus.cc:482] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:18.876135 24847 raft_consensus.cc:504] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.876642 24847 leader_election.cc:288] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41b3f230abf543aab1f0567731bbd5bb (127.21.182.65:45337), a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377)
I20260810 18:06:18.877014 24390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459cd050375f48ce9c9fdad4c38b956d" candidate_uuid: "27ce3ca724224c498ef32a6e8e1c623b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41b3f230abf543aab1f0567731bbd5bb" is_pre_election: true
I20260810 18:06:18.877080 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459cd050375f48ce9c9fdad4c38b956d" candidate_uuid: "27ce3ca724224c498ef32a6e8e1c623b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a614cf9a15b343208e90f06c52901d44" is_pre_election: true
I20260810 18:06:18.877187 24390 raft_consensus.cc:2454] T 459cd050375f48ce9c9fdad4c38b956d P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27ce3ca724224c498ef32a6e8e1c623b in term 0.
I20260810 18:06:18.877274 24544 raft_consensus.cc:2454] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27ce3ca724224c498ef32a6e8e1c623b in term 0.
I20260810 18:06:18.877571 24624 leader_election.cc:302] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ce3ca724224c498ef32a6e8e1c623b, a614cf9a15b343208e90f06c52901d44; no voters:
I20260810 18:06:18.877817 24847 raft_consensus.cc:2781] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:18.877871 24847 raft_consensus.cc:482] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:18.877923 24847 raft_consensus.cc:3037] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:18.879279 24847 raft_consensus.cc:504] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.879700 24847 leader_election.cc:288] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [CANDIDATE]: Term 1 election: Requested vote from peers 41b3f230abf543aab1f0567731bbd5bb (127.21.182.65:45337), a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377)
I20260810 18:06:18.880123 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459cd050375f48ce9c9fdad4c38b956d" candidate_uuid: "27ce3ca724224c498ef32a6e8e1c623b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a614cf9a15b343208e90f06c52901d44"
I20260810 18:06:18.880123 24390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459cd050375f48ce9c9fdad4c38b956d" candidate_uuid: "27ce3ca724224c498ef32a6e8e1c623b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41b3f230abf543aab1f0567731bbd5bb"
I20260810 18:06:18.880265 24390 raft_consensus.cc:3037] T 459cd050375f48ce9c9fdad4c38b956d P 41b3f230abf543aab1f0567731bbd5bb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:18.880265 24544 raft_consensus.cc:3037] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:18.881774 24390 raft_consensus.cc:2454] T 459cd050375f48ce9c9fdad4c38b956d P 41b3f230abf543aab1f0567731bbd5bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ce3ca724224c498ef32a6e8e1c623b in term 1.
I20260810 18:06:18.881773 24544 raft_consensus.cc:2454] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27ce3ca724224c498ef32a6e8e1c623b in term 1.
I20260810 18:06:18.882109 24624 leader_election.cc:302] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27ce3ca724224c498ef32a6e8e1c623b, a614cf9a15b343208e90f06c52901d44; no voters:
I20260810 18:06:18.882355 24847 raft_consensus.cc:2781] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:18.882472 24847 raft_consensus.cc:686] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [term 1 LEADER]: Becoming Leader. State: Replica: 27ce3ca724224c498ef32a6e8e1c623b, State: Running, Role: LEADER
I20260810 18:06:18.882633 24847 consensus_queue.cc:237] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } }
I20260810 18:06:18.884658 24216 catalog_manager.cc:5495] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b reported cstate change: term changed from 0 to 1, leader changed from <none> to 27ce3ca724224c498ef32a6e8e1c623b (127.21.182.67). New cstate: current_term: 1 leader_uuid: "27ce3ca724224c498ef32a6e8e1c623b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27ce3ca724224c498ef32a6e8e1c623b" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34467 } health_report { overall_health: HEALTHY } } }
I20260810 18:06:18.902181 24910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:18.898784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.21.182.67:34467)
0810 18:06:18.899119 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:18.899137 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:18.899147 (+ 10us) server_negotiation.cc:407] Connection header received
0810 18:06:18.899187 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:18.899189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:18.899231 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:18.899343 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:18.899603 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:18.900215 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:18.901085 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:18.901325 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:18.901440 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:18.901703 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:18.901779 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:18.901864 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:18.901958 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 18:06:18.902011 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":138,"threads_started":1}
I20260810 18:06:18.912593 24910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:18.909147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:47271 (local address 127.21.182.67:34467)
0810 18:06:18.909286 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:18.909292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:18.909537 (+ 245us) server_negotiation.cc:407] Connection header received
0810 18:06:18.909624 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:18.909627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:18.909670 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:18.909732 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:18.909946 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:18.910445 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:18.911533 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:18.911737 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:18.911825 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:18.912153 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:18.912223 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:18.912279 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:18.912388 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 18:06:18.912459 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":61}
I20260810 18:06:18.912602 24913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:18.909052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34467 (local address 127.21.182.67:47271)
0810 18:06:18.909388 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:18.909401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:18.909555 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:18.909754 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:18.909756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:18.909763 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:06:18.909849 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:18.909853 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:18.910534 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:18.910536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:18.911410 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:18.911422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:18.911842 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:18.911845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:18.912012 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:18.912041 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:06:18.912313 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:06:18.912320 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:06:18.912379 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 18:06:18.912480 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":4,"thread_start_us":127,"threads_started":1}
I20260810 18:06:18.918509 24544 raft_consensus.cc:1261] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Refusing update from remote peer 27ce3ca724224c498ef32a6e8e1c623b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:18.918663 24390 raft_consensus.cc:1261] T 459cd050375f48ce9c9fdad4c38b956d P 41b3f230abf543aab1f0567731bbd5bb [term 1 FOLLOWER]: Refusing update from remote peer 27ce3ca724224c498ef32a6e8e1c623b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:18.919019 24847 consensus_queue.cc:1035] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a614cf9a15b343208e90f06c52901d44" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:18.919181 24856 consensus_queue.cc:1035] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [LEADER]: Connected to new peer: Peer: permanent_uuid: "41b3f230abf543aab1f0567731bbd5bb" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:06:18.934689 24848 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 18:06:20.500941 24216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:20.501142 24216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:20.503816 24216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e1c23de1ddb467cb2f765742a1dcdbf: Sending DeleteTablet for 3 replicas of tablet 459cd050375f48ce9c9fdad4c38b956d
I20260810 18:06:20.504547 24367 tablet_service.cc:1514] Processing DeleteTablet for tablet 459cd050375f48ce9c9fdad4c38b956d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:20 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:43502
I20260810 18:06:20.504623 24525 tablet_service.cc:1514] Processing DeleteTablet for tablet 459cd050375f48ce9c9fdad4c38b956d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:20 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:45968
I20260810 18:06:20.505026 24938 tablet_replica.cc:330] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44: stopping tablet replica
I20260810 18:06:20.505017 24937 tablet_replica.cc:330] T 459cd050375f48ce9c9fdad4c38b956d P 41b3f230abf543aab1f0567731bbd5bb: stopping tablet replica
I20260810 18:06:20.504994 24676 tablet_service.cc:1514] Processing DeleteTablet for tablet 459cd050375f48ce9c9fdad4c38b956d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:20 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:56306
I20260810 18:06:20.505075 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 24298
I20260810 18:06:20.505174 24938 raft_consensus.cc:2229] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:20.509707 24938 raft_consensus.cc:2258] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:20.510197 24939 tablet_replica.cc:330] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b: stopping tablet replica
I20260810 18:06:20.510399 24939 raft_consensus.cc:2229] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:06:20.510772 24938 ts_tablet_manager.cc:1905] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:20.510879 24939 raft_consensus.cc:2258] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:20.511732 24939 ts_tablet_manager.cc:1905] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:20.513765 24938 ts_tablet_manager.cc:1918] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:06:20.513870 24938 log.cc:1196] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/459cd050375f48ce9c9fdad4c38b956d
I20260810 18:06:20.514185 24938 ts_tablet_manager.cc:1939] T 459cd050375f48ce9c9fdad4c38b956d P a614cf9a15b343208e90f06c52901d44: Deleting consensus metadata
I20260810 18:06:20.514954 24198 catalog_manager.cc:4841] TS a614cf9a15b343208e90f06c52901d44 (127.21.182.66:39377): tablet 459cd050375f48ce9c9fdad4c38b956d (table test-table [id=8224c58a142543758554f08edb293a2c]) successfully deleted
I20260810 18:06:20.515576 24939 ts_tablet_manager.cc:1918] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:06:20.515676 24939 log.cc:1196] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/459cd050375f48ce9c9fdad4c38b956d
I20260810 18:06:20.516067 24939 ts_tablet_manager.cc:1939] T 459cd050375f48ce9c9fdad4c38b956d P 27ce3ca724224c498ef32a6e8e1c623b: Deleting consensus metadata
I20260810 18:06:20.516892 24198 catalog_manager.cc:4841] TS 27ce3ca724224c498ef32a6e8e1c623b (127.21.182.67:34467): tablet 459cd050375f48ce9c9fdad4c38b956d (table test-table [id=8224c58a142543758554f08edb293a2c]) successfully deleted
W20260810 18:06:20.521417 24200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:45337 (error 108)
I20260810 18:06:20.521828 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 24451
I20260810 18:06:20.526437 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:20.525100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:45337 (local address 127.0.0.1:43522)
0810 18:06:20.526199 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:20.526302 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":989,"thread_start_us":346,"threads_started":1}
W20260810 18:06:20.526875 24200 catalog_manager.cc:4584] TS 41b3f230abf543aab1f0567731bbd5bb (127.21.182.65:45337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 459cd050375f48ce9c9fdad4c38b956d: Network error: Client connection negotiation failed: client connection to 127.21.182.65:45337: connect: Connection refused (error 111)
I20260810 18:06:20.533268 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 24607
I20260810 18:06:20.543737 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 24181
2026-08-10T18:06:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7044 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:06:20 dist-test-slave-27sr krb5kdc[24944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:06:20 dist-test-slave-27sr krb5kdc[24944](info): set up 2 sockets
Aug 10 18:06:20 dist-test-slave-27sr krb5kdc[24944](info): commencing operation
krb5kdc: starting...
W20260810 18:06:21.736025 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:06:21 dist-test-slave-27sr krb5kdc[24944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:06:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:06:21.879876 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:40709
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:40709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:21.998868 24962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:21.999285 24962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:21.999380 24962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:22.004148 24962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:22.004299 24962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:22.004349 24962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:22.004395 24962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:22.004442 24962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:22.010828 24962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:40709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:40709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:22.012166 24962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:22.013281 24962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:22.023221 24970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:22.023234 24969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:22.023352 24962 server_base.cc:1034] running on GCE node
W20260810 18:06:22.023238 24972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:22.024325 24962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:22.025588 24962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:22.026805 24962 hybrid_clock.cc:648] HybridClock initialized: now 1786385182026792 us; error 35 us; skew 500 ppm
Aug 10 18:06:22 dist-test-slave-27sr krb5kdc[24944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:22.029902 24962 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:06:22.030892 24962 webserver.cc:460] Webserver started at http://127.21.182.126:44549/ using document root <none> and password file <none>
I20260810 18:06:22.031491 24962 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:22.031587 24962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:22.031837 24962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:22.034019 24962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "c1791a9adbce465da4428dd84536ae71"
format_stamp: "Formatted at 2026-08-10 18:06:22 on dist-test-slave-27sr"
server_key: "d860f370b5ab8ecc996222c0b1c38172"
server_key_iv: "f8d28262ac7cfdf9c8021f2ee2c6c30e"
server_key_version: "encryptionkey@0"
I20260810 18:06:22.034610 24962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "c1791a9adbce465da4428dd84536ae71"
format_stamp: "Formatted at 2026-08-10 18:06:22 on dist-test-slave-27sr"
server_key: "d860f370b5ab8ecc996222c0b1c38172"
server_key_iv: "f8d28262ac7cfdf9c8021f2ee2c6c30e"
server_key_version: "encryptionkey@0"
I20260810 18:06:22.038650 24962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:06:22.041170 24979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:22.042369 24962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:06:22.042518 24962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "c1791a9adbce465da4428dd84536ae71"
format_stamp: "Formatted at 2026-08-10 18:06:22 on dist-test-slave-27sr"
server_key: "d860f370b5ab8ecc996222c0b1c38172"
server_key_iv: "f8d28262ac7cfdf9c8021f2ee2c6c30e"
server_key_version: "encryptionkey@0"
I20260810 18:06:22.042636 24962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:22.061735 24962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:22.065486 24962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:22.065722 24962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:22.073915 24962 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:40709
I20260810 18:06:22.073944 25041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:40709 every 8 connection(s)
I20260810 18:06:22.074991 24962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:22.078081 25042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:22.078145 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 24962
I20260810 18:06:22.078315 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:22.078629 22233 external_mini_cluster.cc:1428] Setting key f24ad95a9f81a4e6b34808ea9be9ab58
I20260810 18:06:22.085731 25042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71: Bootstrap starting.
I20260810 18:06:22.089246 25042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71: Neither blocks nor log segments found. Creating new log.
Aug 10 18:06:22 dist-test-slave-27sr krb5kdc[24944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:22.090360 25042 log.cc:824] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:22.094215 25042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71: No bootstrap required, opened a new log
I20260810 18:06:22.097433 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:22.080288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.21.182.126:40709)
0810 18:06:22.080915 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:22.080934 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:22.080986 (+ 52us) server_negotiation.cc:407] Connection header received
0810 18:06:22.081814 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:22.081839 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:22.082274 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:22.082724 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:22.083711 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:22.084968 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:22.086119 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:22.086835 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:22.087526 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:22.090972 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:22.090978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:22.090994 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:22.091025 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:22.093491 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:22.094041 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:22.094047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:22.094054 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:22.094211 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:22.094613 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:22.094617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:22.094618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:22.095291 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:22.095511 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:22.095839 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 18:06:22.096226 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":148,"threads_started":1}
I20260810 18:06:22.099380 25042 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1791a9adbce465da4428dd84536ae71" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40709 } }
I20260810 18:06:22.099639 25042 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:22.099717 25042 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1791a9adbce465da4428dd84536ae71, State: Initialized, Role: FOLLOWER
I20260810 18:06:22.100329 25042 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1791a9adbce465da4428dd84536ae71" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40709 } }
I20260810 18:06:22.100519 25042 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:22.100593 25042 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:22.100700 25042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:22.101818 25042 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1791a9adbce465da4428dd84536ae71" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40709 } }
I20260810 18:06:22.102241 25042 leader_election.cc:302] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1791a9adbce465da4428dd84536ae71; no voters:
I20260810 18:06:22.102638 25042 leader_election.cc:288] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:22.102833 25048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:22.103067 25048 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [term 1 LEADER]: Becoming Leader. State: Replica: c1791a9adbce465da4428dd84536ae71, State: Running, Role: LEADER
I20260810 18:06:22.103437 25048 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1791a9adbce465da4428dd84536ae71" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40709 } }
I20260810 18:06:22.104278 25042 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:22.105919 25050 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1791a9adbce465da4428dd84536ae71. Latest consensus state: current_term: 1 leader_uuid: "c1791a9adbce465da4428dd84536ae71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1791a9adbce465da4428dd84536ae71" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40709 } } }
I20260810 18:06:22.105962 25049 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1791a9adbce465da4428dd84536ae71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1791a9adbce465da4428dd84536ae71" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40709 } } }
I20260810 18:06:22.106086 25050 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:22.106086 25049 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:22.106602 25058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:22.109969 25058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:22.116221 25058 catalog_manager.cc:1347] Generated new cluster ID: bca9c25dcb0c47dcb89914063d967c36
I20260810 18:06:22.116322 25058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:22.124275 25058 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:22.125306 25058 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:22.130227 25058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1791a9adbce465da4428dd84536ae71: Generated new TSK 0
I20260810 18:06:22.131016 25058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:06:22.164129 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40709
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:22.284729 25071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:22.285084 25071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:22.285130 25071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:22.289520 25071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:22.289613 25071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:22.289676 25071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:22.295511 25071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:22.296854 25071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:22.297963 25071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:22.308416 25076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:22.309554 25077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:22.310108 25071 server_base.cc:1034] running on GCE node
W20260810 18:06:22.310163 25080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:22.310644 25071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:22.312263 25071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:22.313479 25071 hybrid_clock.cc:648] HybridClock initialized: now 1786385182313443 us; error 53 us; skew 500 ppm
Aug 10 18:06:22 dist-test-slave-27sr krb5kdc[24944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:22.316577 25071 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:06:22.317737 25071 webserver.cc:460] Webserver started at http://127.21.182.65:36365/ using document root <none> and password file <none>
I20260810 18:06:22.318397 25071 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:22.318463 25071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:22.318655 25071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:22.321050 25071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "9fccc2afbc004a05bd10c0b09046753b"
format_stamp: "Formatted at 2026-08-10 18:06:22 on dist-test-slave-27sr"
server_key: "028ba02f3c87ae659d9fff3e705eb8a6"
server_key_iv: "849902eff106d33573b9327d6d6634c3"
server_key_version: "encryptionkey@0"
I20260810 18:06:22.321766 25071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "9fccc2afbc004a05bd10c0b09046753b"
format_stamp: "Formatted at 2026-08-10 18:06:22 on dist-test-slave-27sr"
server_key: "028ba02f3c87ae659d9fff3e705eb8a6"
server_key_iv: "849902eff106d33573b9327d6d6634c3"
server_key_version: "encryptionkey@0"
I20260810 18:06:22.326164 25071 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:06:22.329675 25087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:22.331224 25071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:06:22.331393 25071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "9fccc2afbc004a05bd10c0b09046753b"
format_stamp: "Formatted at 2026-08-10 18:06:22 on dist-test-slave-27sr"
server_key: "028ba02f3c87ae659d9fff3e705eb8a6"
server_key_iv: "849902eff106d33573b9327d6d6634c3"
server_key_version: "encryptionkey@0"
I20260810 18:06:22.331507 25071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:22.350982 25071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:22.354784 25071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:22.354983 25071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:22.355674 25071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:22.356727 25071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:22.356781 25071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:22.356827 25071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:22.356853 25071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:22.367349 25071 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:45265
I20260810 18:06:22.367374 25218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:45265 every 8 connection(s)
I20260810 18:06:22.368530 25071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:22.372296 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 25071
I20260810 18:06:22.372435 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:06:22.372789 22233 external_mini_cluster.cc:1428] Setting key 28a18a0516ad844fb7b5d5145a74928c
Aug 10 18:06:22 dist-test-slave-27sr krb5kdc[24944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
I20260810 18:06:22.386309 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:22.370669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:37031 (local address 127.21.182.126:40709)
0810 18:06:22.370857 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:22.370862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:22.372208 (+ 1346us) server_negotiation.cc:407] Connection header received
0810 18:06:22.373561 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:22.373563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:22.373634 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:22.373717 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:22.375444 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:22.376048 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:22.377676 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:22.378077 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:22.378218 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:22.381437 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:22.381445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:22.381452 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:22.381496 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:22.383386 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:22.384273 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:22.384281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:22.384286 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:22.384411 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:22.384926 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:22.384932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:22.384934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:22.385232 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:22.385322 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:22.386038 (+ 716us) server_negotiation.cc:299] Negotiation successful
0810 18:06:22.386174 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 18:06:22.387478 25221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:22.371074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40709 (local address 127.21.182.65:37031)
0810 18:06:22.371982 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:22.372057 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:22.373282 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:22.373884 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:22.373892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:22.374455 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:22.375259 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:22.375286 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:22.376193 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:22.376200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:22.377531 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:22.377543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:22.378255 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:22.378258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:22.378428 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:22.379140 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:22.379160 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:22.381203 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:22.383600 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:22.383606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:22.383623 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:22.384140 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:22.384552 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:22.384555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:22.384556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:22.384749 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:22.385352 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:22.385359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:22.385676 (+ 317us) client_negotiation.cc:769] Sending connection context
0810 18:06:22.385876 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 18:06:22.386335 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
I20260810 18:06:22.389231 25219 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40709
I20260810 18:06:22.389537 25219 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:22.390252 25219 heartbeater.cc:511] Master 127.21.182.126:40709 requested a full tablet report, sending...
I20260810 18:06:22.392269 25000 ts_manager.cc:194] Registered new tserver with Master: 9fccc2afbc004a05bd10c0b09046753b (127.21.182.65:45265)
I20260810 18:06:22.393608 25000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:37031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:06:22.413763 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40709
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:22.630131 25224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:22.630686 25224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:22.630856 25224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:22.636982 25224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:22.637115 25224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:22.637264 25224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:22.645505 25224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:22.647527 25224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:22.649374 25224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:22.666785 25229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:22.666786 25232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:22.666833 25230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:22.667500 25224 server_base.cc:1034] running on GCE node
I20260810 18:06:22.668344 25224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:22.669456 25224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:22.670696 25224 hybrid_clock.cc:648] HybridClock initialized: now 1786385182670621 us; error 96 us; skew 500 ppm
Aug 10 18:06:22 dist-test-slave-27sr krb5kdc[24944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:22.675349 25224 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:06:22.676963 25224 webserver.cc:460] Webserver started at http://127.21.182.66:43997/ using document root <none> and password file <none>
I20260810 18:06:22.677985 25224 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:22.678124 25224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:22.678511 25224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:22.681967 25224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "68790efa892c4d5fae42091b18dff5d4"
format_stamp: "Formatted at 2026-08-10 18:06:22 on dist-test-slave-27sr"
server_key: "8028bb25a9893dce6dbfab7c981c89b7"
server_key_iv: "3a86d496a82ae4150d07136f09e471ba"
server_key_version: "encryptionkey@0"
I20260810 18:06:22.682950 25224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "68790efa892c4d5fae42091b18dff5d4"
format_stamp: "Formatted at 2026-08-10 18:06:22 on dist-test-slave-27sr"
server_key: "8028bb25a9893dce6dbfab7c981c89b7"
server_key_iv: "3a86d496a82ae4150d07136f09e471ba"
server_key_version: "encryptionkey@0"
I20260810 18:06:22.689769 25224 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 18:06:22.694132 25240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:22.696333 25224 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 18:06:22.696565 25224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "68790efa892c4d5fae42091b18dff5d4"
format_stamp: "Formatted at 2026-08-10 18:06:22 on dist-test-slave-27sr"
server_key: "8028bb25a9893dce6dbfab7c981c89b7"
server_key_iv: "3a86d496a82ae4150d07136f09e471ba"
server_key_version: "encryptionkey@0"
I20260810 18:06:22.696753 25224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:22.729851 25224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:22.734843 25224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:22.735186 25224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:22.736232 25224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:22.737666 25224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:22.737766 25224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:22.737936 25224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:22.738010 25224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:22.758790 25224 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:38557
I20260810 18:06:22.758873 25368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:38557 every 8 connection(s)
I20260810 18:06:22.760676 25224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:06:22.770110 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 25224
I20260810 18:06:22.770335 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:06:22.770990 22233 external_mini_cluster.cc:1428] Setting key aa02910f83a317e447958156b236a39d
Aug 10 18:06:22 dist-test-slave-27sr krb5kdc[24944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
I20260810 18:06:22.793262 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:22.764949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:36697 (local address 127.21.182.126:40709)
0810 18:06:22.765394 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:22.765400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:22.768026 (+ 2626us) server_negotiation.cc:407] Connection header received
0810 18:06:22.770423 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:22.770441 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:22.770752 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:22.771022 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:22.774366 (+ 3344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:22.776086 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:22.778317 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:22.778926 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:22.779060 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:22.785207 (+ 6147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:22.785212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:22.785222 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:22.785263 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:22.788579 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:22.789786 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:22.789790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:22.789791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:22.789991 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:22.790751 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:22.790755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:22.790757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:22.791141 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:22.791366 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:22.792551 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0810 18:06:22.792925 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260810 18:06:22.795801 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:22.766300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40709 (local address 127.21.182.66:36697)
0810 18:06:22.767674 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:22.767752 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:22.769856 (+ 2104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:22.771308 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:22.771348 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:22.772445 (+ 1097us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:22.774092 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:22.774146 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:22.776368 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:22.776371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:22.777925 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:22.777945 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:22.779227 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:22.779259 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:22.779853 (+ 594us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:22.781123 (+ 1270us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:22.781179 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:22.784679 (+ 3500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:22.788823 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:22.788855 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:22.788903 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:22.789530 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:22.790198 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:22.790237 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:22.790241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:22.790508 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:22.791355 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:22.791379 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:22.792017 (+ 638us) client_negotiation.cc:769] Sending connection context
0810 18:06:22.792469 (+ 452us) client_negotiation.cc:240] Negotiation successful
0810 18:06:22.793464 (+ 995us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"thread_start_us":191,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
I20260810 18:06:22.798931 25369 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40709
I20260810 18:06:22.799494 25369 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:22.801316 25369 heartbeater.cc:511] Master 127.21.182.126:40709 requested a full tablet report, sending...
I20260810 18:06:22.804973 25000 ts_manager.cc:194] Registered new tserver with Master: 68790efa892c4d5fae42091b18dff5d4 (127.21.182.66:38557)
I20260810 18:06:22.806090 25000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:36697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:06:22.828678 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40709
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:22.984201 25374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:22.984467 25374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:22.984512 25374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:22.988639 25374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:22.988714 25374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:22.988811 25374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:22.994387 25374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:22.995450 25374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:22.996577 25374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:23.006798 25379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:23.006924 25380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:23.006788 25382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:23.007112 25374 server_base.cc:1034] running on GCE node
I20260810 18:06:23.007593 25374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:23.008374 25374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:23.009547 25374 hybrid_clock.cc:648] HybridClock initialized: now 1786385183009487 us; error 72 us; skew 500 ppm
Aug 10 18:06:23 dist-test-slave-27sr krb5kdc[24944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385183, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:23.012883 25374 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:06:23.014019 25374 webserver.cc:460] Webserver started at http://127.21.182.67:46293/ using document root <none> and password file <none>
I20260810 18:06:23.014668 25374 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:23.014761 25374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:23.015008 25374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:23.017480 25374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "a510e3f7c56b48899afeeb5964d9a428"
format_stamp: "Formatted at 2026-08-10 18:06:23 on dist-test-slave-27sr"
server_key: "e6134f7e1fe3aaae15677566eed4e3db"
server_key_iv: "b0519e12ab089a3911782fd856dbb860"
server_key_version: "encryptionkey@0"
I20260810 18:06:23.018114 25374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "a510e3f7c56b48899afeeb5964d9a428"
format_stamp: "Formatted at 2026-08-10 18:06:23 on dist-test-slave-27sr"
server_key: "e6134f7e1fe3aaae15677566eed4e3db"
server_key_iv: "b0519e12ab089a3911782fd856dbb860"
server_key_version: "encryptionkey@0"
I20260810 18:06:23.022334 25374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:06:23.024997 25392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:23.026230 25374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:06:23.026391 25374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "a510e3f7c56b48899afeeb5964d9a428"
format_stamp: "Formatted at 2026-08-10 18:06:23 on dist-test-slave-27sr"
server_key: "e6134f7e1fe3aaae15677566eed4e3db"
server_key_iv: "b0519e12ab089a3911782fd856dbb860"
server_key_version: "encryptionkey@0"
I20260810 18:06:23.026505 25374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:23.039685 25374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:23.043252 25374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:23.043457 25374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:23.044139 25374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:23.045022 25374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:23.045095 25374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:23.045163 25374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:23.045212 25374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:23.055249 25374 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:39743
I20260810 18:06:23.055271 25521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:39743 every 8 connection(s)
I20260810 18:06:23.056329 25374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:06:23.063548 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 25374
I20260810 18:06:23.063676 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:06:23.064064 22233 external_mini_cluster.cc:1428] Setting key cc39655435c980843f4d5f4cc4fec9f1
Aug 10 18:06:23 dist-test-slave-27sr krb5kdc[24944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385183, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:23.070967 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:23.058296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:54411 (local address 127.21.182.126:40709)
0810 18:06:23.058442 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:23.058446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:23.059496 (+ 1050us) server_negotiation.cc:407] Connection header received
0810 18:06:23.060488 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:23.060492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:23.060555 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:23.060662 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:23.061908 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:23.062478 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:23.063871 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:23.064216 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:23.064330 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:23.067310 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:23.067313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:23.067315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:23.067340 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:23.068932 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:23.069547 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:23.069550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:23.069552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:23.069634 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:23.069980 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:23.069983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:23.069984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:23.070163 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:23.070240 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:23.070734 (+ 494us) server_negotiation.cc:299] Negotiation successful
0810 18:06:23.070842 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:06:23.072001 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:23.058668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40709 (local address 127.21.182.67:54411)
0810 18:06:23.059338 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:23.059382 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:23.060266 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:23.060872 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:23.060880 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:23.061245 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:23.061746 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:23.061761 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:23.062617 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:23.062626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:23.063721 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:23.063735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:23.064402 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:23.064407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:23.064569 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:23.065295 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:23.065312 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:23.067149 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:23.069057 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:23.069063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:23.069076 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:23.069440 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:23.069726 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:23.069729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:23.069731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:23.069895 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:23.070269 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:23.070274 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:23.070522 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 18:06:23.070687 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 18:06:23.071036 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":119,"threads_started":1}
I20260810 18:06:23.073331 25522 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40709
I20260810 18:06:23.073576 25522 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:23.074172 25522 heartbeater.cc:511] Master 127.21.182.126:40709 requested a full tablet report, sending...
I20260810 18:06:23.075438 25000 ts_manager.cc:194] Registered new tserver with Master: a510e3f7c56b48899afeeb5964d9a428 (127.21.182.67:39743)
I20260810 18:06:23.076262 25000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:54411
I20260810 18:06:23.078536 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 18:06:23.096525 25046 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:23.094840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.21.182.126:40709)
0810 18:06:23.094983 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:23.094987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:23.095216 (+ 229us) server_negotiation.cc:407] Connection header received
0810 18:06:23.095347 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:23.095350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:23.095399 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:23.095483 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:23.096350 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50186: BlockingRecv error: recv got EOF from 127.0.0.1:50186 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:06:23.098489 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 25071
I20260810 18:06:23.110051 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 25224
I20260810 18:06:23.120044 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 25374
I20260810 18:06:23.129266 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 24962
2026-08-10T18:06:23Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2583 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:06:23 dist-test-slave-27sr krb5kdc[25535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:06:23 dist-test-slave-27sr krb5kdc[25535](info): set up 2 sockets
Aug 10 18:06:23 dist-test-slave-27sr krb5kdc[25535](info): commencing operation
krb5kdc: starting...
W20260810 18:06:24.291126 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:06:24 dist-test-slave-27sr krb5kdc[25535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:06:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:06:24.435555 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:44707
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:36093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:44707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:24.565243 25550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:24.565622 25550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:24.565723 25550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:24.570576 25550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:24.570727 25550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:24.570781 25550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:24.570819 25550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:24.570849 25550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:24.576783 25550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:36093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:44707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:44707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:24.578045 25550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:24.579298 25550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:24.589913 25557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:24.589915 25560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:24.589931 25556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:24.590006 25550 server_base.cc:1034] running on GCE node
I20260810 18:06:24.590890 25550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:24.592351 25550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:24.593581 25550 hybrid_clock.cc:648] HybridClock initialized: now 1786385184593513 us; error 82 us; skew 500 ppm
Aug 10 18:06:24 dist-test-slave-27sr krb5kdc[25535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385184, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:24.597174 25550 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:06:24.598318 25550 webserver.cc:460] Webserver started at http://127.21.182.126:36495/ using document root <none> and password file <none>
I20260810 18:06:24.598954 25550 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:24.599025 25550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:24.599236 25550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:24.601722 25550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "31069226840c4d41951970c371dcfe6f"
format_stamp: "Formatted at 2026-08-10 18:06:24 on dist-test-slave-27sr"
server_key: "a67283897481e6395a1cd83fe02d26dd"
server_key_iv: "98d220eafa725a25f987a36f1e4c3bbf"
server_key_version: "encryptionkey@0"
I20260810 18:06:24.602370 25550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "31069226840c4d41951970c371dcfe6f"
format_stamp: "Formatted at 2026-08-10 18:06:24 on dist-test-slave-27sr"
server_key: "a67283897481e6395a1cd83fe02d26dd"
server_key_iv: "98d220eafa725a25f987a36f1e4c3bbf"
server_key_version: "encryptionkey@0"
I20260810 18:06:24.606990 25550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 18:06:24.610081 25567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:24.611613 25550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:24.611781 25550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "31069226840c4d41951970c371dcfe6f"
format_stamp: "Formatted at 2026-08-10 18:06:24 on dist-test-slave-27sr"
server_key: "a67283897481e6395a1cd83fe02d26dd"
server_key_iv: "98d220eafa725a25f987a36f1e4c3bbf"
server_key_version: "encryptionkey@0"
I20260810 18:06:24.611990 25550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:24.623811 25550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:24.627550 25550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:24.627781 25550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:24.635429 25550 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:44707
I20260810 18:06:24.635855 25626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:44707 every 8 connection(s)
I20260810 18:06:24.636607 25550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:24.639737 25627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:24.644912 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 25550
I20260810 18:06:24.645054 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:24.645349 22233 external_mini_cluster.cc:1428] Setting key 8c58a9a35eabcc137036f215ca070cf7
I20260810 18:06:24.646935 25627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f: Bootstrap starting.
I20260810 18:06:24.649714 25627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:24.650619 25627 log.cc:824] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:24.653026 25627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f: No bootstrap required, opened a new log
Aug 10 18:06:24 dist-test-slave-27sr krb5kdc[25535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:24.657871 25627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31069226840c4d41951970c371dcfe6f" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44707 } }
I20260810 18:06:24.658177 25627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:24.658236 25627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31069226840c4d41951970c371dcfe6f, State: Initialized, Role: FOLLOWER
I20260810 18:06:24.658787 25627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31069226840c4d41951970c371dcfe6f" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44707 } }
I20260810 18:06:24.658946 25627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:24.659039 25627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:24.659159 25627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:24.660224 25627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31069226840c4d41951970c371dcfe6f" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44707 } }
I20260810 18:06:24.660620 25627 leader_election.cc:302] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31069226840c4d41951970c371dcfe6f; no voters:
I20260810 18:06:24.661053 25627 leader_election.cc:288] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:24.661144 25632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:24.661424 25632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [term 1 LEADER]: Becoming Leader. State: Replica: 31069226840c4d41951970c371dcfe6f, State: Running, Role: LEADER
I20260810 18:06:24.661798 25632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31069226840c4d41951970c371dcfe6f" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44707 } }
I20260810 18:06:24.662288 25630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:24.646978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.21.182.126:44707)
0810 18:06:24.647503 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:24.647521 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:24.647559 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:06:24.648326 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:24.648347 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:24.648719 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:24.649034 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:24.650110 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:24.651227 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:24.652271 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:24.652970 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:24.653152 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:24.656325 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:24.656332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:24.656353 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:24.656390 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:24.658471 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:24.659161 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:24.659168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:24.659173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:24.659355 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:24.659687 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:24.659691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:24.659693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:24.660220 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:24.660430 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:24.660755 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 18:06:24.661111 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":139,"threads_started":1}
I20260810 18:06:24.662571 25627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:24.663239 25632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31069226840c4d41951970c371dcfe6f. Latest consensus state: current_term: 1 leader_uuid: "31069226840c4d41951970c371dcfe6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31069226840c4d41951970c371dcfe6f" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44707 } } }
I20260810 18:06:24.663372 25632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:24.663800 25638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:24.664004 25633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31069226840c4d41951970c371dcfe6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31069226840c4d41951970c371dcfe6f" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44707 } } }
I20260810 18:06:24.664181 25633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:24.666028 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:06:24.667292 25638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:24.674196 25638 catalog_manager.cc:1347] Generated new cluster ID: ab63fcb89d5b41d1bcc97d533ba8c9f5
I20260810 18:06:24.674288 25638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:24.689396 25630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:24.683801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47754 (local address 127.21.182.126:44707)
0810 18:06:24.683984 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:24.683989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:24.684283 (+ 294us) server_negotiation.cc:407] Connection header received
0810 18:06:24.684487 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:24.684490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:24.684543 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:24.684634 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:24.685283 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:24.685847 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:24.686763 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:24.687029 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:24.687167 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:24.688256 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:24.688260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:24.688262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:24.688832 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:24.688871 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:24.688890 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:24.689008 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:24.689124 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 18:06:24.689216 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:06:24.702868 25638 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:24.704388 25638 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:24.718175 25638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31069226840c4d41951970c371dcfe6f: Generated new TSK 0
I20260810 18:06:24.719174 25638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:06:24.731817 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 25550
2026-08-10T18:06:24Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1602 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:06:24 dist-test-slave-27sr krb5kdc[25663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:06:24 dist-test-slave-27sr krb5kdc[25663](info): set up 2 sockets
Aug 10 18:06:24 dist-test-slave-27sr krb5kdc[25663](info): commencing operation
krb5kdc: starting...
W20260810 18:06:25.863426 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:06:25 dist-test-slave-27sr krb5kdc[25663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:06:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:06:26.011514 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:40429
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:40429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:26.136487 25676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:26.136868 25676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:26.136945 25676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:26.141258 25676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:26.141361 25676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:26.141402 25676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:26.141460 25676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:26.141497 25676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:26.147387 25676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:40429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:40429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:26.148689 25676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:26.149864 25676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:26.159578 25685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:26.159578 25682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:26.159569 25683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:26.159869 25676 server_base.cc:1034] running on GCE node
I20260810 18:06:26.160631 25676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:26.161964 25676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:26.163206 25676 hybrid_clock.cc:648] HybridClock initialized: now 1786385186163192 us; error 38 us; skew 500 ppm
Aug 10 18:06:26 dist-test-slave-27sr krb5kdc[25663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385186, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:26.170431 25676 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:06:26.171730 25676 webserver.cc:460] Webserver started at http://127.21.182.126:39417/ using document root <none> and password file <none>
I20260810 18:06:26.172508 25676 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:26.172626 25676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:26.172917 25676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:26.175392 25676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "1b68a4f9a82245348239f0b7008dce18"
format_stamp: "Formatted at 2026-08-10 18:06:26 on dist-test-slave-27sr"
server_key: "88e7ca3f61fe63f9605848d340b1132c"
server_key_iv: "ca26a92387d923dad51322c4a457bfdb"
server_key_version: "encryptionkey@0"
I20260810 18:06:26.176157 25676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "1b68a4f9a82245348239f0b7008dce18"
format_stamp: "Formatted at 2026-08-10 18:06:26 on dist-test-slave-27sr"
server_key: "88e7ca3f61fe63f9605848d340b1132c"
server_key_iv: "ca26a92387d923dad51322c4a457bfdb"
server_key_version: "encryptionkey@0"
I20260810 18:06:26.180761 25676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:06:26.183630 25694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:26.185125 25676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:06:26.185303 25676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "1b68a4f9a82245348239f0b7008dce18"
format_stamp: "Formatted at 2026-08-10 18:06:26 on dist-test-slave-27sr"
server_key: "88e7ca3f61fe63f9605848d340b1132c"
server_key_iv: "ca26a92387d923dad51322c4a457bfdb"
server_key_version: "encryptionkey@0"
I20260810 18:06:26.185439 25676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:26.204346 25676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:26.208133 25676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:26.208370 25676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:26.216163 25676 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:40429
I20260810 18:06:26.216468 25756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:40429 every 8 connection(s)
I20260810 18:06:26.217374 25676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:26.220458 25757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:26.220614 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 25676
I20260810 18:06:26.220765 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:26.221086 22233 external_mini_cluster.cc:1428] Setting key a2cde0154bd449d34a7262f96a9b3906
I20260810 18:06:26.227734 25757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18: Bootstrap starting.
I20260810 18:06:26.230633 25757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18: Neither blocks nor log segments found. Creating new log.
Aug 10 18:06:26 dist-test-slave-27sr krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:26.231474 25757 log.cc:824] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:26.234133 25757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18: No bootstrap required, opened a new log
I20260810 18:06:26.236646 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:26.222695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.21.182.126:40429)
0810 18:06:26.223171 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:26.223186 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:26.223222 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:06:26.224120 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:26.224141 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:26.224529 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:26.224888 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:26.225787 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:26.226794 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:26.227700 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:26.228295 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:26.228485 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:26.231425 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:26.231431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:26.231447 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:26.231479 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:26.233569 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:26.234055 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:26.234061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:26.234066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:26.234211 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:26.234506 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:26.234510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:26.234512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:26.234952 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:26.235168 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:26.235476 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 18:06:26.235791 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":131,"threads_started":1}
I20260810 18:06:26.238510 25757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b68a4f9a82245348239f0b7008dce18" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40429 } }
I20260810 18:06:26.238823 25757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:26.238911 25757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b68a4f9a82245348239f0b7008dce18, State: Initialized, Role: FOLLOWER
I20260810 18:06:26.239413 25757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b68a4f9a82245348239f0b7008dce18" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40429 } }
I20260810 18:06:26.239595 25757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:26.239698 25757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:26.239838 25757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:26.240953 25757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b68a4f9a82245348239f0b7008dce18" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40429 } }
I20260810 18:06:26.241341 25757 leader_election.cc:302] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b68a4f9a82245348239f0b7008dce18; no voters:
I20260810 18:06:26.241708 25757 leader_election.cc:288] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:26.241863 25762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:26.242118 25762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [term 1 LEADER]: Becoming Leader. State: Replica: 1b68a4f9a82245348239f0b7008dce18, State: Running, Role: LEADER
I20260810 18:06:26.242441 25762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b68a4f9a82245348239f0b7008dce18" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40429 } }
I20260810 18:06:26.243136 25757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:26.244303 25763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b68a4f9a82245348239f0b7008dce18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b68a4f9a82245348239f0b7008dce18" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40429 } } }
I20260810 18:06:26.244467 25763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:26.244765 25764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b68a4f9a82245348239f0b7008dce18. Latest consensus state: current_term: 1 leader_uuid: "1b68a4f9a82245348239f0b7008dce18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b68a4f9a82245348239f0b7008dce18" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40429 } } }
I20260810 18:06:26.244855 25764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:26.244879 25772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:26.248525 25772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:26.255605 25772 catalog_manager.cc:1347] Generated new cluster ID: b5092c323cb54e9c8a88202f0d66466e
I20260810 18:06:26.255703 25772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:26.269912 25772 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:26.270993 25772 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:26.277052 25772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b68a4f9a82245348239f0b7008dce18: Generated new TSK 0
I20260810 18:06:26.278010 25772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:06:26.319741 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40429
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:26.445773 25784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:26.446126 25784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:26.446213 25784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:26.450625 25784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:26.450742 25784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:26.450843 25784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:26.456548 25784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:26.457844 25784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:26.459004 25784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:26.469930 25790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:26.469903 25791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:26.470012 25784 server_base.cc:1034] running on GCE node
W20260810 18:06:26.469962 25794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:26.470678 25784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:26.471454 25784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:26.472692 25784 hybrid_clock.cc:648] HybridClock initialized: now 1786385186472650 us; error 49 us; skew 500 ppm
Aug 10 18:06:26 dist-test-slave-27sr krb5kdc[25663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385186, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:26.475843 25784 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:06:26.476877 25784 webserver.cc:460] Webserver started at http://127.21.182.65:43159/ using document root <none> and password file <none>
I20260810 18:06:26.477469 25784 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:26.477535 25784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:26.477725 25784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:26.480088 25784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "bdd97eed34a749ce8cd56fc50a8570b7"
format_stamp: "Formatted at 2026-08-10 18:06:26 on dist-test-slave-27sr"
server_key: "19d4c7ad6ebcba4c6b007655b54ae6d5"
server_key_iv: "36db81360e3d9ffeb2467dfe14adaff4"
server_key_version: "encryptionkey@0"
I20260810 18:06:26.480677 25784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "bdd97eed34a749ce8cd56fc50a8570b7"
format_stamp: "Formatted at 2026-08-10 18:06:26 on dist-test-slave-27sr"
server_key: "19d4c7ad6ebcba4c6b007655b54ae6d5"
server_key_iv: "36db81360e3d9ffeb2467dfe14adaff4"
server_key_version: "encryptionkey@0"
I20260810 18:06:26.484828 25784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:06:26.487425 25803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:26.488986 25784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:06:26.489128 25784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "bdd97eed34a749ce8cd56fc50a8570b7"
format_stamp: "Formatted at 2026-08-10 18:06:26 on dist-test-slave-27sr"
server_key: "19d4c7ad6ebcba4c6b007655b54ae6d5"
server_key_iv: "36db81360e3d9ffeb2467dfe14adaff4"
server_key_version: "encryptionkey@0"
I20260810 18:06:26.489270 25784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:26.502072 25784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:26.505867 25784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:26.506102 25784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:26.506785 25784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:26.507720 25784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:26.507771 25784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:26.507848 25784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:26.507879 25784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:26.517694 25784 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:46305
I20260810 18:06:26.517714 25930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:46305 every 8 connection(s)
I20260810 18:06:26.518713 25784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:26.519588 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 25784
I20260810 18:06:26.519833 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:06:26.520212 22233 external_mini_cluster.cc:1428] Setting key 33feed8744969066412a5c7f9f60ccff
Aug 10 18:06:26 dist-test-slave-27sr krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385186, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
I20260810 18:06:26.536239 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:26.521282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:60671 (local address 127.21.182.126:40429)
0810 18:06:26.521455 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:26.521461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:26.522464 (+ 1003us) server_negotiation.cc:407] Connection header received
0810 18:06:26.524880 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:26.524886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:26.524952 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:26.525053 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:26.526313 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:26.527271 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:26.528582 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:26.529030 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:26.529155 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:26.532115 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:26.532119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:26.532121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:26.532150 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:26.533690 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:26.534283 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:26.534286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:26.534288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:26.534374 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:26.534796 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:26.534803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:26.534808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:26.535105 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:26.535220 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:26.535982 (+ 762us) server_negotiation.cc:299] Negotiation successful
0810 18:06:26.536104 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:06:26.537499 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:26.521716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40429 (local address 127.21.182.65:60671)
0810 18:06:26.522280 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:26.522337 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:26.524631 (+ 2294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:26.525222 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:26.525229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:26.525629 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:26.526142 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:26.526156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:26.527438 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:26.527442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:26.528447 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:26.528457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:26.529192 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:26.529195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:26.529351 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:26.530050 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:26.530068 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:26.531881 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:26.533849 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:26.533857 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:26.533883 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:26.534164 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:26.534461 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:26.534464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:26.534465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:26.534671 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:26.535232 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:26.535240 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:26.535612 (+ 372us) client_negotiation.cc:769] Sending connection context
0810 18:06:26.535858 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 18:06:26.536320 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":161,"threads_started":1}
I20260810 18:06:26.539016 25931 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40429
I20260810 18:06:26.539309 25931 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:26.540028 25931 heartbeater.cc:511] Master 127.21.182.126:40429 requested a full tablet report, sending...
I20260810 18:06:26.541934 25712 ts_manager.cc:194] Registered new tserver with Master: bdd97eed34a749ce8cd56fc50a8570b7 (127.21.182.65:46305)
I20260810 18:06:26.543292 25712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:60671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:06:26.554844 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40429
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:26.683430 25936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:26.683784 25936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:26.683892 25936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:26.688272 25936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:26.688364 25936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:26.688476 25936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:26.694578 25936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:26.695873 25936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:26.697106 25936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:26.708036 25941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:26.708030 25942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:26.708030 25944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:26.708500 25936 server_base.cc:1034] running on GCE node
I20260810 18:06:26.708961 25936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:26.709831 25936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:26.711038 25936 hybrid_clock.cc:648] HybridClock initialized: now 1786385186710992 us; error 54 us; skew 500 ppm
Aug 10 18:06:26 dist-test-slave-27sr krb5kdc[25663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385186, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:26.714587 25936 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:06:26.715725 25936 webserver.cc:460] Webserver started at http://127.21.182.66:40551/ using document root <none> and password file <none>
I20260810 18:06:26.716456 25936 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:26.716560 25936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:26.716842 25936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:26.719185 25936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "08bb4f1565ce4353ad798635478db9ac"
format_stamp: "Formatted at 2026-08-10 18:06:26 on dist-test-slave-27sr"
server_key: "4feff2660f5e3ee566d190cf50aaf0b7"
server_key_iv: "2d6865fcbc043d3552d06b08b7b89e94"
server_key_version: "encryptionkey@0"
I20260810 18:06:26.719827 25936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "08bb4f1565ce4353ad798635478db9ac"
format_stamp: "Formatted at 2026-08-10 18:06:26 on dist-test-slave-27sr"
server_key: "4feff2660f5e3ee566d190cf50aaf0b7"
server_key_iv: "2d6865fcbc043d3552d06b08b7b89e94"
server_key_version: "encryptionkey@0"
I20260810 18:06:26.724326 25936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:06:26.727312 25954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:26.728931 25936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:26.729094 25936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "08bb4f1565ce4353ad798635478db9ac"
format_stamp: "Formatted at 2026-08-10 18:06:26 on dist-test-slave-27sr"
server_key: "4feff2660f5e3ee566d190cf50aaf0b7"
server_key_iv: "2d6865fcbc043d3552d06b08b7b89e94"
server_key_version: "encryptionkey@0"
I20260810 18:06:26.729210 25936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:26.751256 25936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:26.755095 25936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:26.755338 25936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:26.756050 25936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:26.757014 25936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:26.757104 25936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:26.757184 25936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:26.757217 25936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:26.767107 25936 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:39559
I20260810 18:06:26.767985 26081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:39559 every 8 connection(s)
I20260810 18:06:26.768534 25936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:06:26.774215 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 25936
I20260810 18:06:26.774325 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:06:26.774609 22233 external_mini_cluster.cc:1428] Setting key 65c5d84c257414cf4cfbbae57a80da9d
Aug 10 18:06:26 dist-test-slave-27sr krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385186, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:26.785386 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:26.770750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:52661 (local address 127.21.182.126:40429)
0810 18:06:26.770951 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:26.770956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:26.772071 (+ 1115us) server_negotiation.cc:407] Connection header received
0810 18:06:26.773477 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:26.773480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:26.773535 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:26.773616 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:26.775047 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:26.775793 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:26.777608 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:26.777842 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:26.777936 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:26.780956 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:26.780959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:26.780961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:26.780988 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:26.782615 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:26.783408 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:26.783412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:26.783413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:26.783503 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:26.784003 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:26.784006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:26.784007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:26.784224 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:26.784336 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:26.785115 (+ 779us) server_negotiation.cc:299] Negotiation successful
0810 18:06:26.785244 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 18:06:26.786759 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:26.771314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40429 (local address 127.21.182.66:52661)
0810 18:06:26.771841 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:26.771942 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:26.773250 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:26.773793 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:26.773801 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:26.774219 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:26.774875 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:26.774892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:26.776009 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:26.776013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:26.777467 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:26.777485 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:26.778017 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:26.778024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:26.778278 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:26.779110 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:26.779127 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:26.780740 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:26.782773 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:26.782784 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:26.782809 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:26.783252 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:26.783612 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:26.783618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:26.783622 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:26.783851 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:26.784379 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:26.784390 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:26.784783 (+ 393us) client_negotiation.cc:769] Sending connection context
0810 18:06:26.785029 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 18:06:26.785470 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
I20260810 18:06:26.788456 26082 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40429
I20260810 18:06:26.788801 26082 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:26.789458 26082 heartbeater.cc:511] Master 127.21.182.126:40429 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
I20260810 18:06:26.790946 25712 ts_manager.cc:194] Registered new tserver with Master: 08bb4f1565ce4353ad798635478db9ac (127.21.182.66:39559)
I20260810 18:06:26.791590 25712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:52661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:06:26.810940 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40429
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:26.941270 26088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:26.941629 26088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:26.941741 26088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:26.946969 26088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:26.947088 26088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:26.947190 26088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:26.953760 26088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:26.955065 26088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:26.956373 26088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:26.967545 26094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:26.967588 26093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:26.967545 26096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:26.967646 26088 server_base.cc:1034] running on GCE node
I20260810 18:06:26.968318 26088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:26.969050 26088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:26.970261 26088 hybrid_clock.cc:648] HybridClock initialized: now 1786385186970241 us; error 45 us; skew 500 ppm
Aug 10 18:06:26 dist-test-slave-27sr krb5kdc[25663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385186, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:26.973297 26088 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:06:26.974397 26088 webserver.cc:460] Webserver started at http://127.21.182.67:42879/ using document root <none> and password file <none>
I20260810 18:06:26.975008 26088 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:26.975108 26088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:26.975370 26088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:26.977566 26088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "285bfef8be604b4786e815e28f6d7f34"
format_stamp: "Formatted at 2026-08-10 18:06:26 on dist-test-slave-27sr"
server_key: "0d27b55921fc1bffa9332705c62bcddf"
server_key_iv: "30bad5332b3faf8c5e778f397a5cb35c"
server_key_version: "encryptionkey@0"
I20260810 18:06:26.978201 26088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "285bfef8be604b4786e815e28f6d7f34"
format_stamp: "Formatted at 2026-08-10 18:06:26 on dist-test-slave-27sr"
server_key: "0d27b55921fc1bffa9332705c62bcddf"
server_key_iv: "30bad5332b3faf8c5e778f397a5cb35c"
server_key_version: "encryptionkey@0"
I20260810 18:06:26.982362 26088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:06:26.985075 26104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:26.986420 26088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:26.986586 26088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "285bfef8be604b4786e815e28f6d7f34"
format_stamp: "Formatted at 2026-08-10 18:06:26 on dist-test-slave-27sr"
server_key: "0d27b55921fc1bffa9332705c62bcddf"
server_key_iv: "30bad5332b3faf8c5e778f397a5cb35c"
server_key_version: "encryptionkey@0"
I20260810 18:06:26.986697 26088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:26.999294 26088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:27.002943 26088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:27.003170 26088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:27.003816 26088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:27.004823 26088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:27.004897 26088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:27.004974 26088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:27.005017 26088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:27.014961 26088 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:41795
I20260810 18:06:27.014983 26234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:41795 every 8 connection(s)
I20260810 18:06:27.016040 26088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:06:27.019423 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 26088
I20260810 18:06:27.019560 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:06:27.019973 22233 external_mini_cluster.cc:1428] Setting key 270d9f730bd631d583190d2fec01e7f5
Aug 10 18:06:27 dist-test-slave-27sr krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385186, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:27.032182 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:27.018093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:40831 (local address 127.21.182.126:40429)
0810 18:06:27.018283 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:27.018289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:27.019245 (+ 956us) server_negotiation.cc:407] Connection header received
0810 18:06:27.021365 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:27.021368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:27.021432 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:27.021529 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:27.023246 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:27.023844 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:27.025276 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:27.025494 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:27.025579 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:27.028321 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:27.028325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:27.028327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:27.028355 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:27.030081 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:27.030579 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:27.030581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:27.030583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:27.030668 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:27.031077 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:27.031079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:27.031080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:27.031270 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:27.031346 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:27.031872 (+ 526us) server_negotiation.cc:299] Negotiation successful
0810 18:06:27.032050 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 18:06:27.033356 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:27.018509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40429 (local address 127.21.182.67:40831)
0810 18:06:27.019019 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:27.019067 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:27.021101 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:27.021850 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:27.021857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:27.022491 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:27.023079 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:27.023093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:27.024204 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:27.024207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:27.025148 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:27.025158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:27.025640 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:27.025643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:27.025794 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:27.026432 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:27.026450 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:27.028124 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:27.030187 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:27.030193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:27.030208 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:27.030484 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:27.030813 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:27.030815 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:27.030816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:27.030993 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:27.031361 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:27.031368 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:27.031641 (+ 273us) client_negotiation.cc:769] Sending connection context
0810 18:06:27.031814 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 18:06:27.032203 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":136,"threads_started":1}
I20260810 18:06:27.034814 26235 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40429
I20260810 18:06:27.035092 26235 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:27.035637 26235 heartbeater.cc:511] Master 127.21.182.126:40429 requested a full tablet report, sending...
I20260810 18:06:27.036835 25712 ts_manager.cc:194] Registered new tserver with Master: 285bfef8be604b4786e815e28f6d7f34 (127.21.182.67:41795)
I20260810 18:06:27.037436 25712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:40831
I20260810 18:06:27.046736 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:06:27 dist-test-slave-27sr krb5kdc[25663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385187, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 18:06:27 dist-test-slave-27sr krb5kdc[25663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385187, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:27.074646 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:27.065651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.21.182.126:40429)
0810 18:06:27.065835 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:27.065839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:27.065896 (+ 57us) server_negotiation.cc:407] Connection header received
0810 18:06:27.066021 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:27.066023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:27.066070 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:27.066159 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:27.066932 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:27.067499 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:27.068453 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:27.068674 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:27.068772 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:27.071330 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:27.071334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:27.071336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:27.071366 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:27.073060 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:27.073489 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:27.073492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:27.073493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:27.073574 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:27.073814 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:27.073817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:27.073818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:27.074064 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:27.074168 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:27.074403 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 18:06:27.074513 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260810 18:06:27.075168 25712 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52442
I20260810 18:06:27.076110 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 25784
I20260810 18:06:27.087122 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 25936
I20260810 18:06:27.097807 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 26088
I20260810 18:06:27.107625 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 25676
2026-08-10T18:06:27Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2377 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:06:27 dist-test-slave-27sr krb5kdc[26248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:06:27 dist-test-slave-27sr krb5kdc[26248](info): set up 2 sockets
Aug 10 18:06:27 dist-test-slave-27sr krb5kdc[26248](info): commencing operation
krb5kdc: starting...
W20260810 18:06:28.310982 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:06:28 dist-test-slave-27sr krb5kdc[26248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:06:28.447651 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:35997
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:35997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:28.565984 26262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:28.566322 26262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:28.566404 26262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:28.570633 26262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:28.570740 26262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:28.570781 26262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:28.570842 26262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:28.570879 26262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:28.576526 26262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:35997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:35997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:28.577677 26262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:28.578800 26262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:28.587829 26270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:28.587829 26268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:28.588052 26262 server_base.cc:1034] running on GCE node
W20260810 18:06:28.587852 26267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:28.588791 26262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:28.589947 26262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:28.591122 26262 hybrid_clock.cc:648] HybridClock initialized: now 1786385188591112 us; error 31 us; skew 500 ppm
Aug 10 18:06:28 dist-test-slave-27sr krb5kdc[26248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385188, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:28.594292 26262 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:06:28.595321 26262 webserver.cc:460] Webserver started at http://127.21.182.126:37361/ using document root <none> and password file <none>
I20260810 18:06:28.596006 26262 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:28.596107 26262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:28.596395 26262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:28.599452 26262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "73fec726324f4d9aa0e782cc4a3bd884"
format_stamp: "Formatted at 2026-08-10 18:06:28 on dist-test-slave-27sr"
server_key: "ba22f1d2e553df2ebd8186ffcc79a5bd"
server_key_iv: "952456edd693ceb8377375b4d01b1732"
server_key_version: "encryptionkey@0"
I20260810 18:06:28.600214 26262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "73fec726324f4d9aa0e782cc4a3bd884"
format_stamp: "Formatted at 2026-08-10 18:06:28 on dist-test-slave-27sr"
server_key: "ba22f1d2e553df2ebd8186ffcc79a5bd"
server_key_iv: "952456edd693ceb8377375b4d01b1732"
server_key_version: "encryptionkey@0"
I20260810 18:06:28.604460 26262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:06:28.607003 26279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:28.608382 26262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:06:28.608529 26262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "73fec726324f4d9aa0e782cc4a3bd884"
format_stamp: "Formatted at 2026-08-10 18:06:28 on dist-test-slave-27sr"
server_key: "ba22f1d2e553df2ebd8186ffcc79a5bd"
server_key_iv: "952456edd693ceb8377375b4d01b1732"
server_key_version: "encryptionkey@0"
I20260810 18:06:28.608646 26262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:28.618469 26262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:28.621881 26262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:28.622089 26262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:28.629516 26262 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:35997
I20260810 18:06:28.629545 26341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:35997 every 8 connection(s)
I20260810 18:06:28.630476 26262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:28.633316 26343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:28.634902 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 26262
I20260810 18:06:28.635030 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:28.635314 22233 external_mini_cluster.cc:1428] Setting key 9008dbf8cf79f50497abacd5e6538f97
I20260810 18:06:28.640322 26343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884: Bootstrap starting.
I20260810 18:06:28.642999 26343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:28.643967 26343 log.cc:824] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:06:28 dist-test-slave-27sr krb5kdc[26248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:28.646435 26343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884: No bootstrap required, opened a new log
I20260810 18:06:28.651005 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:28.636888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.21.182.126:35997)
0810 18:06:28.637336 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:28.637358 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:28.637414 (+ 56us) server_negotiation.cc:407] Connection header received
0810 18:06:28.638271 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:28.638297 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:28.638744 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:28.639126 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:28.639815 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:28.641285 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:28.642232 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:28.642921 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:28.643101 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:28.646191 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:28.646197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:28.646211 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:28.646239 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:28.648038 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:28.648574 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:28.648579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:28.648583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:28.648693 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:28.649020 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:28.649023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:28.649024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:28.649411 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:28.649569 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:28.649895 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 18:06:28.650228 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260810 18:06:28.650910 26343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fec726324f4d9aa0e782cc4a3bd884" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35997 } }
I20260810 18:06:28.651151 26343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:28.651211 26343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73fec726324f4d9aa0e782cc4a3bd884, State: Initialized, Role: FOLLOWER
I20260810 18:06:28.651681 26343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fec726324f4d9aa0e782cc4a3bd884" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35997 } }
I20260810 18:06:28.651800 26343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:28.651841 26343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:28.651969 26343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:28.652921 26343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fec726324f4d9aa0e782cc4a3bd884" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35997 } }
I20260810 18:06:28.653311 26343 leader_election.cc:302] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73fec726324f4d9aa0e782cc4a3bd884; no voters:
I20260810 18:06:28.653663 26343 leader_election.cc:288] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:28.653786 26348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:28.654054 26348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [term 1 LEADER]: Becoming Leader. State: Replica: 73fec726324f4d9aa0e782cc4a3bd884, State: Running, Role: LEADER
I20260810 18:06:28.654358 26348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fec726324f4d9aa0e782cc4a3bd884" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35997 } }
I20260810 18:06:28.654979 26343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:28.655984 26349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73fec726324f4d9aa0e782cc4a3bd884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fec726324f4d9aa0e782cc4a3bd884" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35997 } } }
I20260810 18:06:28.656123 26349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:28.656493 26350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73fec726324f4d9aa0e782cc4a3bd884. Latest consensus state: current_term: 1 leader_uuid: "73fec726324f4d9aa0e782cc4a3bd884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fec726324f4d9aa0e782cc4a3bd884" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35997 } } }
I20260810 18:06:28.656590 26350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:28.656939 26357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:28.659796 26357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:28.665328 26357 catalog_manager.cc:1347] Generated new cluster ID: 334e5eea78b845b8b13601d44ef0cb48
I20260810 18:06:28.665400 26357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:28.679428 26357 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:28.680815 26357 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:28.685792 26357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73fec726324f4d9aa0e782cc4a3bd884: Generated new TSK 0
I20260810 18:06:28.686604 26357 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:06:28.720193 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35997
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:28.849901 26372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:28.850235 26372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:28.850333 26372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:28.854669 26372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:28.854779 26372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:28.854873 26372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:28.860792 26372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:28.862042 26372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:28.863148 26372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:28.873437 26378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:28.873437 26379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:28.873677 26372 server_base.cc:1034] running on GCE node
W20260810 18:06:28.873570 26381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:28.874344 26372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:28.875063 26372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:28.876257 26372 hybrid_clock.cc:648] HybridClock initialized: now 1786385188876230 us; error 37 us; skew 500 ppm
Aug 10 18:06:28 dist-test-slave-27sr krb5kdc[26248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385188, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:28.879335 26372 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:06:28.880321 26372 webserver.cc:460] Webserver started at http://127.21.182.65:44689/ using document root <none> and password file <none>
I20260810 18:06:28.880920 26372 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:28.881017 26372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:28.881263 26372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:28.883462 26372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "e62c98dfca1c4e1ab5f3583908a965c3"
format_stamp: "Formatted at 2026-08-10 18:06:28 on dist-test-slave-27sr"
server_key: "926d426b34e6b39ebe468cc643c8ab9e"
server_key_iv: "90b5f231cdc763538448f9d45e1096b9"
server_key_version: "encryptionkey@0"
I20260810 18:06:28.884147 26372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "e62c98dfca1c4e1ab5f3583908a965c3"
format_stamp: "Formatted at 2026-08-10 18:06:28 on dist-test-slave-27sr"
server_key: "926d426b34e6b39ebe468cc643c8ab9e"
server_key_iv: "90b5f231cdc763538448f9d45e1096b9"
server_key_version: "encryptionkey@0"
I20260810 18:06:28.888085 26372 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:06:28.890554 26388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:28.891860 26372 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:06:28.892051 26372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "e62c98dfca1c4e1ab5f3583908a965c3"
format_stamp: "Formatted at 2026-08-10 18:06:28 on dist-test-slave-27sr"
server_key: "926d426b34e6b39ebe468cc643c8ab9e"
server_key_iv: "90b5f231cdc763538448f9d45e1096b9"
server_key_version: "encryptionkey@0"
I20260810 18:06:28.892169 26372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:28.910844 26372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:28.914439 26372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:28.914678 26372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:28.915323 26372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:28.916270 26372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:28.916344 26372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:28.916416 26372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:28.916468 26372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:28.925691 26372 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:36553
I20260810 18:06:28.925704 26522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:36553 every 8 connection(s)
I20260810 18:06:28.926661 26372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:28.928537 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 26372
I20260810 18:06:28.928701 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:06:28.929041 22233 external_mini_cluster.cc:1428] Setting key b84768411ecc99b4946ca6ec69e281b4
Aug 10 18:06:28 dist-test-slave-27sr krb5kdc[26248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385188, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
I20260810 18:06:28.944819 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:28.928814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:58645 (local address 127.21.182.126:35997)
0810 18:06:28.928991 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:28.928996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:28.930131 (+ 1135us) server_negotiation.cc:407] Connection header received
0810 18:06:28.931951 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:28.931957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:28.932039 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:28.932127 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:28.934098 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:28.935218 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:28.936849 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:28.937128 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:28.937240 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:28.940825 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:28.940829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:28.940831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:28.940857 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:28.942608 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:28.943169 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:28.943175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:28.943177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:28.943298 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:28.943641 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:28.943644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:28.943645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:28.943839 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:28.943982 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:28.944560 (+ 578us) server_negotiation.cc:299] Negotiation successful
0810 18:06:28.944686 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
I20260810 18:06:28.946017 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:28.929440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35997 (local address 127.21.182.65:58645)
0810 18:06:28.929961 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:28.930016 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:28.931631 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:28.932437 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:28.932449 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:28.933018 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:28.933849 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:28.933876 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:28.935382 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:28.935386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:28.936668 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:28.936679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:28.937503 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:28.937508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:28.937732 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:28.938481 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:28.938504 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:28.940610 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:28.942729 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:28.942734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:28.942748 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:28.943023 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:28.943405 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:28.943408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:28.943410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:28.943553 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:28.943987 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:28.943999 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:28.944262 (+ 263us) client_negotiation.cc:769] Sending connection context
0810 18:06:28.944518 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 18:06:28.944841 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":159,"threads_started":1}
I20260810 18:06:28.948766 26523 heartbeater.cc:347] Connected to a master server at 127.21.182.126:35997
I20260810 18:06:28.949066 26523 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:28.949724 26523 heartbeater.cc:511] Master 127.21.182.126:35997 requested a full tablet report, sending...
I20260810 18:06:28.951521 26299 ts_manager.cc:194] Registered new tserver with Master: e62c98dfca1c4e1ab5f3583908a965c3 (127.21.182.65:36553)
I20260810 18:06:28.952787 26299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:58645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:06:28.964339 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35997
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:29.083217 26529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:29.083488 26529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:29.083550 26529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:29.087728 26529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:29.087798 26529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:29.087898 26529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:29.093499 26529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:29.094758 26529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:29.096032 26529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:29.107372 26536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:29.107406 26539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:29.107372 26534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:29.107698 26529 server_base.cc:1034] running on GCE node
I20260810 18:06:29.108328 26529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:29.109217 26529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:29.110447 26529 hybrid_clock.cc:648] HybridClock initialized: now 1786385189110426 us; error 33 us; skew 500 ppm
Aug 10 18:06:29 dist-test-slave-27sr krb5kdc[26248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:29.113708 26529 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:06:29.114718 26529 webserver.cc:460] Webserver started at http://127.21.182.66:34449/ using document root <none> and password file <none>
I20260810 18:06:29.115309 26529 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:29.115375 26529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:29.115556 26529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:29.117883 26529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "d11683ccd646492685baa00fde8c676a"
format_stamp: "Formatted at 2026-08-10 18:06:29 on dist-test-slave-27sr"
server_key: "5b5926ec6834e97dccd224dd80c41ede"
server_key_iv: "69ff5401f5e57c6228db7fe1eddb28e2"
server_key_version: "encryptionkey@0"
I20260810 18:06:29.118464 26529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "d11683ccd646492685baa00fde8c676a"
format_stamp: "Formatted at 2026-08-10 18:06:29 on dist-test-slave-27sr"
server_key: "5b5926ec6834e97dccd224dd80c41ede"
server_key_iv: "69ff5401f5e57c6228db7fe1eddb28e2"
server_key_version: "encryptionkey@0"
I20260810 18:06:29.122641 26529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 18:06:29.125411 26546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:29.126714 26529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:06:29.126875 26529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "d11683ccd646492685baa00fde8c676a"
format_stamp: "Formatted at 2026-08-10 18:06:29 on dist-test-slave-27sr"
server_key: "5b5926ec6834e97dccd224dd80c41ede"
server_key_iv: "69ff5401f5e57c6228db7fe1eddb28e2"
server_key_version: "encryptionkey@0"
I20260810 18:06:29.127002 26529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:29.166222 26529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:29.169778 26529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:29.170037 26529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:29.170776 26529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:29.171811 26529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:29.171898 26529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:29.172020 26529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:29.172083 26529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:29.182489 26529 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:42159
I20260810 18:06:29.182542 26674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:42159 every 8 connection(s)
I20260810 18:06:29.183609 26529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:06:29.193010 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 26529
I20260810 18:06:29.193172 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:06:29.193539 22233 external_mini_cluster.cc:1428] Setting key 71730cc6421ec357e6f80ef7aaee34f4
Aug 10 18:06:29 dist-test-slave-27sr krb5kdc[26248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:29.200412 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:29.185942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:55021 (local address 127.21.182.126:35997)
0810 18:06:29.186107 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:29.186112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:29.187147 (+ 1035us) server_negotiation.cc:407] Connection header received
0810 18:06:29.188746 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:29.188750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:29.188808 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:29.188896 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:29.190239 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:29.190911 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:29.192221 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:29.192446 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:29.192545 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:29.195988 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:29.195992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:29.195994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:29.196021 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:29.197816 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:29.198447 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:29.198455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:29.198457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:29.198569 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:29.198943 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:29.198948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:29.198953 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:29.199254 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:29.199344 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:29.199967 (+ 623us) server_negotiation.cc:299] Negotiation successful
0810 18:06:29.200144 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:06:29.201303 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:29.186437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35997 (local address 127.21.182.66:55021)
0810 18:06:29.186956 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:29.187005 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:29.188488 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:29.189102 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:29.189109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:29.189514 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:29.190051 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:29.190066 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:29.191048 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:29.191051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:29.192092 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:29.192103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:29.192607 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:29.192610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:29.192859 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:29.193603 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:29.193630 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:29.195740 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:29.197987 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:29.197994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:29.198011 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:29.198319 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:29.198677 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:29.198679 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:29.198681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:29.198838 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:29.199380 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:29.199387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:29.199668 (+ 281us) client_negotiation.cc:769] Sending connection context
0810 18:06:29.199850 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 18:06:29.200241 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":128,"threads_started":1}
I20260810 18:06:29.202715 26675 heartbeater.cc:347] Connected to a master server at 127.21.182.126:35997
I20260810 18:06:29.203001 26675 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:29.203549 26675 heartbeater.cc:511] Master 127.21.182.126:35997 requested a full tablet report, sending...
I20260810 18:06:29.204782 26299 ts_manager.cc:194] Registered new tserver with Master: d11683ccd646492685baa00fde8c676a (127.21.182.66:42159)
I20260810 18:06:29.205430 26299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:55021
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:06:29.226850 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35997
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:29.349696 26680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:29.350015 26680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:29.350063 26680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:29.354369 26680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:29.354444 26680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:29.354578 26680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:29.360391 26680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:29.361626 26680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:29.362762 26680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:29.372924 26688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:29.373011 26685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:29.373015 26686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:29.373314 26680 server_base.cc:1034] running on GCE node
I20260810 18:06:29.373723 26680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:29.374372 26680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:29.375569 26680 hybrid_clock.cc:648] HybridClock initialized: now 1786385189375549 us; error 31 us; skew 500 ppm
Aug 10 18:06:29 dist-test-slave-27sr krb5kdc[26248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:29.378564 26680 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:06:29.379519 26680 webserver.cc:460] Webserver started at http://127.21.182.67:40563/ using document root <none> and password file <none>
I20260810 18:06:29.380167 26680 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:29.380234 26680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:29.380425 26680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:29.382539 26680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "79a7595a86bc47bca33f22dc5b0a917f"
format_stamp: "Formatted at 2026-08-10 18:06:29 on dist-test-slave-27sr"
server_key: "f5976efe696d92689d1a4af77d4f9bb9"
server_key_iv: "b75f251486c10ebb283981d67ce6023a"
server_key_version: "encryptionkey@0"
I20260810 18:06:29.383102 26680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "79a7595a86bc47bca33f22dc5b0a917f"
format_stamp: "Formatted at 2026-08-10 18:06:29 on dist-test-slave-27sr"
server_key: "f5976efe696d92689d1a4af77d4f9bb9"
server_key_iv: "b75f251486c10ebb283981d67ce6023a"
server_key_version: "encryptionkey@0"
I20260810 18:06:29.387192 26680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:06:29.389811 26695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:29.391214 26680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:29.391356 26680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "79a7595a86bc47bca33f22dc5b0a917f"
format_stamp: "Formatted at 2026-08-10 18:06:29 on dist-test-slave-27sr"
server_key: "f5976efe696d92689d1a4af77d4f9bb9"
server_key_iv: "b75f251486c10ebb283981d67ce6023a"
server_key_version: "encryptionkey@0"
I20260810 18:06:29.391506 26680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:29.406617 26680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:29.410032 26680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:29.410259 26680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:29.410940 26680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:29.411978 26680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:29.412042 26680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:29.412115 26680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:29.412154 26680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:29.422076 26680 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:42353
I20260810 18:06:29.422137 26823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:42353 every 8 connection(s)
I20260810 18:06:29.423266 26680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:06:29.425420 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 26680
I20260810 18:06:29.425575 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:06:29.425885 22233 external_mini_cluster.cc:1428] Setting key dfbd44d44347b842b73060dd5765b193
Aug 10 18:06:29 dist-test-slave-27sr krb5kdc[26248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:29.438946 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:29.425462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:55351 (local address 127.21.182.126:35997)
0810 18:06:29.425647 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:29.425653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:29.426659 (+ 1006us) server_negotiation.cc:407] Connection header received
0810 18:06:29.428422 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:29.428425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:29.428474 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:29.428541 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:29.429839 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:29.430472 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:29.431761 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:29.432011 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:29.432096 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:29.434916 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:29.434924 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:29.434927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:29.434954 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:29.436570 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:29.437179 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:29.437183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:29.437185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:29.437277 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:29.437675 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:29.437679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:29.437680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:29.437895 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:29.438007 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:29.438615 (+ 608us) server_negotiation.cc:299] Negotiation successful
0810 18:06:29.438768 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 18:06:29.440148 26826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:29.425945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35997 (local address 127.21.182.67:55351)
0810 18:06:29.426506 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:29.426566 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:29.428184 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:29.428738 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:29.428745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:29.429169 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:29.429702 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:29.429716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:29.430597 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:29.430599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:29.431638 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:29.431649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:29.432158 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:29.432161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:29.432326 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:29.432974 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:29.432992 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:29.434707 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:29.436721 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:29.436729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:29.436749 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:29.437063 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:29.437379 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:29.437386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:29.437388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:29.437562 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:29.438008 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:29.438019 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:29.438347 (+ 328us) client_negotiation.cc:769] Sending connection context
0810 18:06:29.438572 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 18:06:29.438996 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":166,"threads_started":1}
I20260810 18:06:29.441314 26824 heartbeater.cc:347] Connected to a master server at 127.21.182.126:35997
I20260810 18:06:29.441562 26824 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:29.442094 26824 heartbeater.cc:511] Master 127.21.182.126:35997 requested a full tablet report, sending...
I20260810 18:06:29.443218 26299 ts_manager.cc:194] Registered new tserver with Master: 79a7595a86bc47bca33f22dc5b0a917f (127.21.182.67:42353)
I20260810 18:06:29.443836 26299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:55351
I20260810 18:06:29.452610 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:06:29 dist-test-slave-27sr krb5kdc[26248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:06:29 dist-test-slave-27sr krb5kdc[26248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.65@KRBTEST.COM
I20260810 18:06:29.482108 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:29.471863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59844 (local address 127.21.182.65:36553)
0810 18:06:29.472260 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:29.472267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:29.472293 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:06:29.472349 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:29.472355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:29.472570 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:29.472705 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:29.473497 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:29.474282 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:29.475122 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:29.475526 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:29.475616 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:29.478297 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:29.478308 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:29.478326 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:29.478355 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:29.480165 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:29.480596 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:29.480601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:29.480603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:29.480693 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:29.480999 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:29.481001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:29.481003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:29.481279 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:29.481449 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:29.481693 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 18:06:29.481891 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
Aug 10 18:06:29 dist-test-slave-27sr krb5kdc[26248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:29.494699 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:29.485373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.21.182.126:35997)
0810 18:06:29.485558 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:29.485562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:29.485577 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:06:29.485708 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:29.485710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:29.485763 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:29.485831 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:29.486426 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:29.486993 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:29.488140 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:29.488355 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:29.488449 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:29.491338 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:29.491341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:29.491343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:29.491369 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:29.493010 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:29.493443 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:29.493446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:29.493448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:29.493532 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:29.493785 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:29.493787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:29.493788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:29.494038 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:29.494147 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:29.494371 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 18:06:29.494481 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 18:06:29.495198 26299 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35154
I20260810 18:06:29.496054 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 26372
I20260810 18:06:29.507385 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 26529
I20260810 18:06:29.518379 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 26680
I20260810 18:06:29.528043 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 26262
2026-08-10T18:06:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2419 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:06:29 dist-test-slave-27sr krb5kdc[26836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:06:29 dist-test-slave-27sr krb5kdc[26836](info): set up 2 sockets
Aug 10 18:06:29 dist-test-slave-27sr krb5kdc[26836](info): commencing operation
krb5kdc: starting...
W20260810 18:06:30.679253 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260810 18:06:30.913666 22233 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.133s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/test-user.keytab.
W20260810 18:06:31.513298 22233 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.600s user 0.001s sys 0.001s
Aug 10 18:06:31 dist-test-slave-27sr krb5kdc[26836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:06:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
W20260810 18:06:32.332257 22233 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.182.126: real 0.791s user 0.000s sys 0.004s
I20260810 18:06:32.332851 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:43063
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:43063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:32.460536 26850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:32.460857 26850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:32.460959 26850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:32.465250 26850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:32.465346 26850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:32.465394 26850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:32.465452 26850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:32.465493 26850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:32.471586 26850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:43063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:43063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:32.472918 26850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:32.474184 26850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:32.483613 26859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:32.483626 26856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:32.483706 26857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:32.483863 26850 server_base.cc:1034] running on GCE node
I20260810 18:06:32.484584 26850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:32.485977 26850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:32.487167 26850 hybrid_clock.cc:648] HybridClock initialized: now 1786385192487142 us; error 38 us; skew 500 ppm
Aug 10 18:06:32 dist-test-slave-27sr krb5kdc[26836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385192, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:32.495460 26850 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:06:32.496417 26850 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:32.496519 26850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:32.496807 26850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:32.499131 26850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "393432dc26d2420aa8fc993ac98543a7"
format_stamp: "Formatted at 2026-08-10 18:06:32 on dist-test-slave-27sr"
server_key: "324778d8809d87c16d29cccf37b94fa7"
server_key_iv: "046810b726032b09f411280628529744"
server_key_version: "encryptionkey@0"
I20260810 18:06:32.499768 26850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "393432dc26d2420aa8fc993ac98543a7"
format_stamp: "Formatted at 2026-08-10 18:06:32 on dist-test-slave-27sr"
server_key: "324778d8809d87c16d29cccf37b94fa7"
server_key_iv: "046810b726032b09f411280628529744"
server_key_version: "encryptionkey@0"
I20260810 18:06:32.504272 26850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:06:32.507074 26865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:32.508626 26850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:32.508775 26850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "393432dc26d2420aa8fc993ac98543a7"
format_stamp: "Formatted at 2026-08-10 18:06:32 on dist-test-slave-27sr"
server_key: "324778d8809d87c16d29cccf37b94fa7"
server_key_iv: "046810b726032b09f411280628529744"
server_key_version: "encryptionkey@0"
I20260810 18:06:32.508898 26850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:32.534559 26850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:32.538024 26850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:32.538255 26850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:32.546334 26850 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:43063
I20260810 18:06:32.546638 26925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:43063 every 8 connection(s)
I20260810 18:06:32.547111 26850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:32.550068 26926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:32.551569 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 26850
I20260810 18:06:32.551715 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:32.552073 22233 external_mini_cluster.cc:1428] Setting key 186d52f2aab7adeb4703e6e51d93658d
I20260810 18:06:32.557053 26926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7: Bootstrap starting.
I20260810 18:06:32.560034 26926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:32.561072 26926 log.cc:824] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:06:32 dist-test-slave-27sr krb5kdc[26836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:32.563613 26926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7: No bootstrap required, opened a new log
I20260810 18:06:32.567329 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:32.553569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.21.182.126:43063)
0810 18:06:32.553984 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:32.554006 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:32.554058 (+ 52us) server_negotiation.cc:407] Connection header received
0810 18:06:32.554712 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:32.554730 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:32.555075 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:32.555360 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:32.556346 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:32.557415 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:32.558596 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:32.559076 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:32.559195 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:32.562295 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:32.562301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:32.562317 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:32.562344 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:32.564247 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:32.564783 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:32.564788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:32.564793 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:32.564915 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:32.565207 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:32.565209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:32.565210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:32.565643 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:32.565792 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:32.566157 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 18:06:32.566464 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":129,"threads_started":1}
I20260810 18:06:32.568147 26926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "393432dc26d2420aa8fc993ac98543a7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43063 } }
I20260810 18:06:32.568416 26926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:32.568503 26926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 393432dc26d2420aa8fc993ac98543a7, State: Initialized, Role: FOLLOWER
I20260810 18:06:32.568977 26926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "393432dc26d2420aa8fc993ac98543a7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43063 } }
I20260810 18:06:32.569139 26926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:32.569228 26926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:32.569355 26926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:32.570639 26926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "393432dc26d2420aa8fc993ac98543a7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43063 } }
I20260810 18:06:32.571017 26926 leader_election.cc:302] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 393432dc26d2420aa8fc993ac98543a7; no voters:
I20260810 18:06:32.571364 26926 leader_election.cc:288] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:32.571529 26934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:32.571769 26934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [term 1 LEADER]: Becoming Leader. State: Replica: 393432dc26d2420aa8fc993ac98543a7, State: Running, Role: LEADER
I20260810 18:06:32.572140 26934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "393432dc26d2420aa8fc993ac98543a7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43063 } }
I20260810 18:06:32.572842 26926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:32.574067 26935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 393432dc26d2420aa8fc993ac98543a7. Latest consensus state: current_term: 1 leader_uuid: "393432dc26d2420aa8fc993ac98543a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "393432dc26d2420aa8fc993ac98543a7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43063 } } }
I20260810 18:06:32.574267 26935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:32.574069 26936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "393432dc26d2420aa8fc993ac98543a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "393432dc26d2420aa8fc993ac98543a7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43063 } } }
I20260810 18:06:32.574386 26936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7 [sys.catalog]: This master's current role is: LEADER
W20260810 18:06:32.578033 26952 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:06:32.578121 26952 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:06:32.578222 26944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:32.578995 26944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:32.585011 26944 catalog_manager.cc:1347] Generated new cluster ID: d8c832145fa6458ca9a7ffaf329b90d3
I20260810 18:06:32.585109 26944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:32.605028 26944 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:32.606621 26944 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:32.613013 26944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7: Generated new TSK 0
I20260810 18:06:32.613878 26944 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:06:32.647408 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43063
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:32.770881 26959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:32.771216 26959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:32.771306 26959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:32.775758 26959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:32.775857 26959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:32.775979 26959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:32.781564 26959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:32.782783 26959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:32.783931 26959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:32.795688 26966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:32.795676 26968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:32.795667 26965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:32.795754 26959 server_base.cc:1034] running on GCE node
I20260810 18:06:32.796389 26959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:32.797111 26959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:32.798273 26959 hybrid_clock.cc:648] HybridClock initialized: now 1786385192798261 us; error 31 us; skew 500 ppm
Aug 10 18:06:32 dist-test-slave-27sr krb5kdc[26836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385192, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:32.801400 26959 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:06:32.802294 26959 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:32.802399 26959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:32.802646 26959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:32.805066 26959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "7c27b8e999b149ebbff718a5d598126c"
format_stamp: "Formatted at 2026-08-10 18:06:32 on dist-test-slave-27sr"
server_key: "c752cba365e01157e522e0a87f8c458a"
server_key_iv: "133f09a5cf9077a69cbbcb83277c1d9e"
server_key_version: "encryptionkey@0"
I20260810 18:06:32.805672 26959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "7c27b8e999b149ebbff718a5d598126c"
format_stamp: "Formatted at 2026-08-10 18:06:32 on dist-test-slave-27sr"
server_key: "c752cba365e01157e522e0a87f8c458a"
server_key_iv: "133f09a5cf9077a69cbbcb83277c1d9e"
server_key_version: "encryptionkey@0"
I20260810 18:06:32.809801 26959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 18:06:32.812597 26974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:32.813997 26959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:06:32.814147 26959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "7c27b8e999b149ebbff718a5d598126c"
format_stamp: "Formatted at 2026-08-10 18:06:32 on dist-test-slave-27sr"
server_key: "c752cba365e01157e522e0a87f8c458a"
server_key_iv: "133f09a5cf9077a69cbbcb83277c1d9e"
server_key_version: "encryptionkey@0"
I20260810 18:06:32.814275 26959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:32.831123 26959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:32.834287 26959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:32.834527 26959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:32.835163 26959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:32.836160 26959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:32.836246 26959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:32.836396 26959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:32.836462 26959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:32.845952 26959 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:37983
I20260810 18:06:32.845973 27102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:37983 every 8 connection(s)
I20260810 18:06:32.846654 26959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:32.856145 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 26959
I20260810 18:06:32.856313 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:06:32.856602 22233 external_mini_cluster.cc:1428] Setting key ed78e1894fca3b7dcf08ca8255a66fa0
Aug 10 18:06:32 dist-test-slave-27sr krb5kdc[26836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385192, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:32.862115 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:32.848684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:35461 (local address 127.21.182.126:43063)
0810 18:06:32.848835 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:32.848839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:32.850123 (+ 1284us) server_negotiation.cc:407] Connection header received
0810 18:06:32.851064 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:32.851069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:32.851132 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:32.851237 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:32.852540 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:32.853081 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:32.854333 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:32.854570 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:32.854687 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:32.857640 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:32.857643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:32.857645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:32.857670 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:32.859526 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:32.860183 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:32.860190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:32.860195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:32.860297 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:32.860768 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:32.860774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:32.860776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:32.861022 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:32.861097 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:32.861701 (+ 604us) server_negotiation.cc:299] Negotiation successful
0810 18:06:32.861869 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:06:32.862772 27106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:32.849084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:43063 (local address 127.21.182.65:35461)
0810 18:06:32.849956 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:32.849998 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:32.850836 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:32.851435 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:32.851442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:32.851840 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:32.852380 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:32.852394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:32.853210 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:32.853218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:32.854195 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:32.854205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:32.854734 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:32.854738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:32.854911 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:32.855493 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:32.855510 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:32.857481 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:32.859658 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:32.859667 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:32.859687 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:32.860073 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:32.860414 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:32.860420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:32.860422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:32.860632 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:32.861159 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:32.861164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:32.861417 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 18:06:32.861577 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 18:06:32.861851 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":133,"threads_started":1}
I20260810 18:06:32.864154 27104 heartbeater.cc:347] Connected to a master server at 127.21.182.126:43063
I20260810 18:06:32.864441 27104 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:32.865059 27104 heartbeater.cc:511] Master 127.21.182.126:43063 requested a full tablet report, sending...
I20260810 18:06:32.866698 26884 ts_manager.cc:194] Registered new tserver with Master: 7c27b8e999b149ebbff718a5d598126c (127.21.182.65:37983)
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
I20260810 18:06:32.867900 26884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:35461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:06:32.888211 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43063
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:33.011754 27109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:33.012121 27109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:33.012173 27109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:33.016489 27109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:33.016559 27109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:33.016623 27109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:33.022228 27109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:33.023351 27109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:33.024497 27109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:33.034739 27120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:33.034739 27117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:33.034739 27116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:33.035002 27109 server_base.cc:1034] running on GCE node
I20260810 18:06:33.035557 27109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:33.036441 27109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:33.037777 27109 hybrid_clock.cc:648] HybridClock initialized: now 1786385193037731 us; error 62 us; skew 500 ppm
Aug 10 18:06:33 dist-test-slave-27sr krb5kdc[26836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385193, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:33.041253 27109 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:06:33.042039 27109 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:33.042107 27109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:33.042296 27109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:33.044576 27109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "2c0464dff5be4a72b251132611e89e2a"
format_stamp: "Formatted at 2026-08-10 18:06:33 on dist-test-slave-27sr"
server_key: "4b654c8a1ac8f4409b96b70e8e427706"
server_key_iv: "50198e3b2045995df2211b2a7b282789"
server_key_version: "encryptionkey@0"
I20260810 18:06:33.045163 27109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "2c0464dff5be4a72b251132611e89e2a"
format_stamp: "Formatted at 2026-08-10 18:06:33 on dist-test-slave-27sr"
server_key: "4b654c8a1ac8f4409b96b70e8e427706"
server_key_iv: "50198e3b2045995df2211b2a7b282789"
server_key_version: "encryptionkey@0"
I20260810 18:06:33.049508 27109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:06:33.052188 27126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:33.053619 27109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:33.053783 27109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "2c0464dff5be4a72b251132611e89e2a"
format_stamp: "Formatted at 2026-08-10 18:06:33 on dist-test-slave-27sr"
server_key: "4b654c8a1ac8f4409b96b70e8e427706"
server_key_iv: "50198e3b2045995df2211b2a7b282789"
server_key_version: "encryptionkey@0"
I20260810 18:06:33.053920 27109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:33.073442 27109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:33.077644 27109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:33.077827 27109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:33.078429 27109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:33.079339 27109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:33.079399 27109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:33.079442 27109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:33.079458 27109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:33.088796 27109 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:46533
I20260810 18:06:33.088815 27255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:46533 every 8 connection(s)
I20260810 18:06:33.089515 27109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:06:33.096652 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 27109
I20260810 18:06:33.096768 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:06:33.097095 22233 external_mini_cluster.cc:1428] Setting key 614f66a030e2de6ab1bc9d24a4685d2c
Aug 10 18:06:33 dist-test-slave-27sr krb5kdc[26836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385193, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:33.105630 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.091485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:32969 (local address 127.21.182.126:43063)
0810 18:06:33.091649 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:33.091652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:33.092996 (+ 1344us) server_negotiation.cc:407] Connection header received
0810 18:06:33.093883 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:33.093886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:33.093936 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:33.094018 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:33.095254 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.095773 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.097162 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.097475 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.097625 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.100679 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:33.100686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:33.100692 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:33.100732 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:33.102529 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:33.103326 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:33.103329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:33.103330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:33.103423 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:33.104036 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:33.104040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:33.104041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:33.104293 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:33.104416 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:33.105335 (+ 919us) server_negotiation.cc:299] Negotiation successful
0810 18:06:33.105494 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:06:33.107223 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.091878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:43063 (local address 127.21.182.66:32969)
0810 18:06:33.092820 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:33.092874 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:33.093685 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:33.094188 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:33.094195 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:33.094578 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:33.095111 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.095126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.095968 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.095971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.097007 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.097018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.097679 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.097683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.097871 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.098579 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:33.098597 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:33.100467 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:33.102681 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:33.102692 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:33.102720 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:33.103196 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:33.103530 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:33.103537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:33.103543 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:33.103790 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:33.104454 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:33.104466 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:33.104947 (+ 481us) client_negotiation.cc:769] Sending connection context
0810 18:06:33.105246 (+ 299us) client_negotiation.cc:240] Negotiation successful
0810 18:06:33.105738 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":123,"threads_started":1}
I20260810 18:06:33.108898 27257 heartbeater.cc:347] Connected to a master server at 127.21.182.126:43063
I20260810 18:06:33.109206 27257 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:33.109776 27257 heartbeater.cc:511] Master 127.21.182.126:43063 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
I20260810 18:06:33.111063 26884 ts_manager.cc:194] Registered new tserver with Master: 2c0464dff5be4a72b251132611e89e2a (127.21.182.66:46533)
I20260810 18:06:33.111744 26884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:32969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:06:33.131695 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43063
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:33.255358 27262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:33.255669 27262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:33.255717 27262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:33.260242 27262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:33.260326 27262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:33.260429 27262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:33.266247 27262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:33.267524 27262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:33.268694 27262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:33.279304 27269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:33.279292 27272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:33.279287 27270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:33.279606 27262 server_base.cc:1034] running on GCE node
I20260810 18:06:33.280157 27262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:33.280912 27262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:33.282092 27262 hybrid_clock.cc:648] HybridClock initialized: now 1786385193282072 us; error 33 us; skew 500 ppm
Aug 10 18:06:33 dist-test-slave-27sr krb5kdc[26836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385193, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:33.285239 27262 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:06:33.286091 27262 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:33.286161 27262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:33.286342 27262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:33.288615 27262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "a56409990ef6421a98edd2e11a9f8279"
format_stamp: "Formatted at 2026-08-10 18:06:33 on dist-test-slave-27sr"
server_key: "75e7420ec56be7a76a5954a12dda17b6"
server_key_iv: "7965510c1116f5d29b1ce382948bfbe2"
server_key_version: "encryptionkey@0"
I20260810 18:06:33.289206 27262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "a56409990ef6421a98edd2e11a9f8279"
format_stamp: "Formatted at 2026-08-10 18:06:33 on dist-test-slave-27sr"
server_key: "75e7420ec56be7a76a5954a12dda17b6"
server_key_iv: "7965510c1116f5d29b1ce382948bfbe2"
server_key_version: "encryptionkey@0"
I20260810 18:06:33.293419 27262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:06:33.296181 27278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:33.297607 27262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:33.297734 27262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "a56409990ef6421a98edd2e11a9f8279"
format_stamp: "Formatted at 2026-08-10 18:06:33 on dist-test-slave-27sr"
server_key: "75e7420ec56be7a76a5954a12dda17b6"
server_key_iv: "7965510c1116f5d29b1ce382948bfbe2"
server_key_version: "encryptionkey@0"
I20260810 18:06:33.297847 27262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:33.309710 27262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:33.313474 27262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:33.313655 27262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:33.314262 27262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:33.315172 27262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:33.315224 27262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:33.315268 27262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:33.315290 27262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:33.325721 27262 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:41805
I20260810 18:06:33.325749 27407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:41805 every 8 connection(s)
I20260810 18:06:33.326470 27262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:06:33.330159 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 27262
I20260810 18:06:33.330325 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:06:33.330677 22233 external_mini_cluster.cc:1428] Setting key 5fcd6824ef41cd8d40737e8b07f03d9c
Aug 10 18:06:33 dist-test-slave-27sr krb5kdc[26836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385193, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:33.342010 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.328627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:57783 (local address 127.21.182.126:43063)
0810 18:06:33.328786 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:33.328791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:33.330275 (+ 1484us) server_negotiation.cc:407] Connection header received
0810 18:06:33.331522 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:33.331526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:33.331588 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:33.331678 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:33.333477 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.334044 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.335201 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.335449 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.335530 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.338452 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:33.338456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:33.338458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:33.338486 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:33.339883 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:33.340489 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:33.340493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:33.340495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:33.340598 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:33.340947 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:33.340949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:33.340950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:33.341148 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:33.341229 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:33.341746 (+ 517us) server_negotiation.cc:299] Negotiation successful
0810 18:06:33.341876 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:06:33.342958 27410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.329080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:43063 (local address 127.21.182.67:57783)
0810 18:06:33.330039 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:33.330105 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:33.331229 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:33.332006 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:33.332016 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:33.332583 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:33.333310 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.333333 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.334160 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.334163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.335088 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.335097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.335588 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.335590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.335744 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.336513 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:33.336532 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:33.338242 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:33.340066 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:33.340071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:33.340087 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:33.340381 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:33.340686 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:33.340688 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:33.340689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:33.340863 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:33.341239 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:33.341245 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:33.341518 (+ 273us) client_negotiation.cc:769] Sending connection context
0810 18:06:33.341698 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 18:06:33.342024 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":118,"threads_started":1}
I20260810 18:06:33.344477 27408 heartbeater.cc:347] Connected to a master server at 127.21.182.126:43063
I20260810 18:06:33.344797 27408 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:33.345412 27408 heartbeater.cc:511] Master 127.21.182.126:43063 requested a full tablet report, sending...
I20260810 18:06:33.346517 26884 ts_manager.cc:194] Registered new tserver with Master: a56409990ef6421a98edd2e11a9f8279 (127.21.182.67:41805)
I20260810 18:06:33.347092 26884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:57783
I20260810 18:06:33.357635 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:06:33.367715 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.359993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.21.182.126:43063)
0810 18:06:33.360131 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:33.360135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:33.360262 (+ 127us) server_negotiation.cc:407] Connection header received
0810 18:06:33.360384 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:33.360387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:33.360434 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:33.360517 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:33.361319 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.361855 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.362665 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.362899 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.362980 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.364562 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:33.364565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:33.364567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:33.364593 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:33.366127 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:33.366551 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:33.366554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:33.366555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:33.366636 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:33.366873 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:33.366875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:33.366876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:33.367124 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:33.367237 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:33.367481 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 18:06:33.367586 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:06:33.371018 26884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:06:33.374749 26884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:33.390479 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.384301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:37983 (local address 127.0.0.1:46684)
0810 18:06:33.384860 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:33.384919 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:33.385075 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:33.386045 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:33.386052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:33.386076 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:33.386230 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.386239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.387514 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.387520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.388743 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.388753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.389849 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.389853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.389966 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.390006 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 18:06:33.390133 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 18:06:33.390276 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":93,"threads_started":1}
I20260810 18:06:33.390502 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.384470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.21.182.65:37983)
0810 18:06:33.385434 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:33.385443 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:33.385471 (+ 28us) server_negotiation.cc:407] Connection header received
0810 18:06:33.385556 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:33.385565 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:33.385808 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:33.385991 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:33.386399 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.387399 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.388888 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.389515 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.389622 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.389782 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:33.390216 (+ 434us) server_negotiation.cc:299] Negotiation successful
0810 18:06:33.390340 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":142,"threads_started":1}
I20260810 18:06:33.391049 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.384489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:46533 (local address 127.0.0.1:53438)
0810 18:06:33.385062 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:33.385080 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:33.385174 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:33.386363 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:33.386365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:33.386382 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:33.386485 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.386491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.388033 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.388036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.389314 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.389333 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.390617 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.390621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.390715 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.390734 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:06:33.390842 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 18:06:33.390915 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":35,"thread_start_us":102,"threads_started":1}
I20260810 18:06:33.391160 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.385183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.21.182.66:46533)
0810 18:06:33.385680 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:33.385694 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:33.385718 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:06:33.385798 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:33.385803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:33.386004 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:33.386168 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:33.386620 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.387846 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.389472 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.390303 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.390409 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.390545 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:33.390884 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 18:06:33.390994 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":159,"threads_started":1}
I20260810 18:06:33.391539 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.385657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52818 (local address 127.21.182.67:41805)
0810 18:06:33.386330 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:33.386346 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:33.386370 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:06:33.386445 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:33.386449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:33.386633 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:33.386797 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:33.387022 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.388163 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.389598 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.390207 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.390316 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.390464 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:33.391268 (+ 804us) server_negotiation.cc:299] Negotiation successful
0810 18:06:33.391391 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":161,"threads_started":1}
I20260810 18:06:33.391607 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.385181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:41805 (local address 127.0.0.1:52818)
0810 18:06:33.386159 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:33.386172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:33.386254 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:33.386789 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:33.386793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:33.386821 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:33.386898 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.386904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.388307 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.388311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.389472 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.389480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.390542 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.390547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.390806 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.390833 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:06:33.390905 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 18:06:33.390981 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"mutex_wait_us":27,"thread_start_us":132,"threads_started":1}
I20260810 18:06:33.394052 27180 tablet_service.cc:1467] Processing CreateTablet for tablet 59a21d19defb4245b005b56a2917ad67 (DEFAULT_TABLE table=test-table [id=a78a32f0b9db4b899e2a8a05967595d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:33.394155 27330 tablet_service.cc:1467] Processing CreateTablet for tablet 59a21d19defb4245b005b56a2917ad67 (DEFAULT_TABLE table=test-table [id=a78a32f0b9db4b899e2a8a05967595d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:33.394147 27032 tablet_service.cc:1467] Processing CreateTablet for tablet 59a21d19defb4245b005b56a2917ad67 (DEFAULT_TABLE table=test-table [id=a78a32f0b9db4b899e2a8a05967595d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:33.395200 27180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a21d19defb4245b005b56a2917ad67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:33.395200 27032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a21d19defb4245b005b56a2917ad67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:33.395208 27330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a21d19defb4245b005b56a2917ad67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:33.402098 27426 tablet_bootstrap.cc:492] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a: Bootstrap starting.
I20260810 18:06:33.403602 27427 tablet_bootstrap.cc:492] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c: Bootstrap starting.
I20260810 18:06:33.404227 27428 tablet_bootstrap.cc:492] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279: Bootstrap starting.
I20260810 18:06:33.404942 27426 tablet_bootstrap.cc:654] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:33.406111 27428 tablet_bootstrap.cc:654] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:33.406107 27427 tablet_bootstrap.cc:654] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:33.406211 27426 log.cc:824] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:33.406989 27427 log.cc:824] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:33.407264 27428 log.cc:824] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:33.408730 27426 tablet_bootstrap.cc:492] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a: No bootstrap required, opened a new log
I20260810 18:06:33.408949 27426 ts_tablet_manager.cc:1397] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:06:33.409411 27427 tablet_bootstrap.cc:492] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c: No bootstrap required, opened a new log
I20260810 18:06:33.409428 27428 tablet_bootstrap.cc:492] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279: No bootstrap required, opened a new log
I20260810 18:06:33.409602 27427 ts_tablet_manager.cc:1397] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:06:33.409632 27428 ts_tablet_manager.cc:1397] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:06:33.412734 27428 raft_consensus.cc:348] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c27b8e999b149ebbff718a5d598126c" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37983 } } peers { permanent_uuid: "2c0464dff5be4a72b251132611e89e2a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 46533 } } peers { permanent_uuid: "a56409990ef6421a98edd2e11a9f8279" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 41805 } }
I20260810 18:06:33.413049 27428 raft_consensus.cc:374] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:33.413116 27428 raft_consensus.cc:729] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a56409990ef6421a98edd2e11a9f8279, State: Initialized, Role: FOLLOWER
I20260810 18:06:33.413167 27426 raft_consensus.cc:348] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c27b8e999b149ebbff718a5d598126c" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37983 } } peers { permanent_uuid: "2c0464dff5be4a72b251132611e89e2a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 46533 } } peers { permanent_uuid: "a56409990ef6421a98edd2e11a9f8279" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 41805 } }
I20260810 18:06:33.413455 27426 raft_consensus.cc:374] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:33.413513 27426 raft_consensus.cc:729] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c0464dff5be4a72b251132611e89e2a, State: Initialized, Role: FOLLOWER
I20260810 18:06:33.413645 27428 consensus_queue.cc:260] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c27b8e999b149ebbff718a5d598126c" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37983 } } peers { permanent_uuid: "2c0464dff5be4a72b251132611e89e2a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 46533 } } peers { permanent_uuid: "a56409990ef6421a98edd2e11a9f8279" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 41805 } }
I20260810 18:06:33.413961 27426 consensus_queue.cc:260] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c27b8e999b149ebbff718a5d598126c" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37983 } } peers { permanent_uuid: "2c0464dff5be4a72b251132611e89e2a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 46533 } } peers { permanent_uuid: "a56409990ef6421a98edd2e11a9f8279" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 41805 } }
I20260810 18:06:33.414124 27427 raft_consensus.cc:348] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c27b8e999b149ebbff718a5d598126c" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37983 } } peers { permanent_uuid: "2c0464dff5be4a72b251132611e89e2a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 46533 } } peers { permanent_uuid: "a56409990ef6421a98edd2e11a9f8279" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 41805 } }
I20260810 18:06:33.414364 27427 raft_consensus.cc:374] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:33.414423 27427 raft_consensus.cc:729] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c27b8e999b149ebbff718a5d598126c, State: Initialized, Role: FOLLOWER
I20260810 18:06:33.414829 27428 ts_tablet_manager.cc:1428] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:06:33.414872 27427 consensus_queue.cc:260] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c27b8e999b149ebbff718a5d598126c" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37983 } } peers { permanent_uuid: "2c0464dff5be4a72b251132611e89e2a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 46533 } } peers { permanent_uuid: "a56409990ef6421a98edd2e11a9f8279" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 41805 } }
I20260810 18:06:33.414996 27426 ts_tablet_manager.cc:1428] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:06:33.415270 27408 heartbeater.cc:503] Master 127.21.182.126:43063 was elected leader, sending a full tablet report...
I20260810 18:06:33.415571 27257 heartbeater.cc:503] Master 127.21.182.126:43063 was elected leader, sending a full tablet report...
I20260810 18:06:33.415794 27427 ts_tablet_manager.cc:1428] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:06:33.416186 27104 heartbeater.cc:503] Master 127.21.182.126:43063 was elected leader, sending a full tablet report...
I20260810 18:06:33.522771 27435 raft_consensus.cc:482] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:33.523079 27435 raft_consensus.cc:504] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c27b8e999b149ebbff718a5d598126c" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37983 } } peers { permanent_uuid: "2c0464dff5be4a72b251132611e89e2a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 46533 } } peers { permanent_uuid: "a56409990ef6421a98edd2e11a9f8279" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 41805 } }
I20260810 18:06:33.524432 27435 leader_election.cc:288] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c0464dff5be4a72b251132611e89e2a (127.21.182.66:46533), a56409990ef6421a98edd2e11a9f8279 (127.21.182.67:41805)
I20260810 18:06:33.528313 27434 raft_consensus.cc:482] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:33.528534 27434 raft_consensus.cc:504] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c27b8e999b149ebbff718a5d598126c" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37983 } } peers { permanent_uuid: "2c0464dff5be4a72b251132611e89e2a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 46533 } } peers { permanent_uuid: "a56409990ef6421a98edd2e11a9f8279" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 41805 } }
I20260810 18:06:33.528721 27437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.524758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:41805 (local address 127.21.182.65:60861)
0810 18:06:33.525144 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:33.525177 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:33.525325 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:33.525620 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:33.525623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:33.525639 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:33.525753 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.525758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.526499 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.526504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.527505 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.527515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.528223 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.528226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.528329 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.528357 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:06:33.528453 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 18:06:33.528534 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":78,"thread_start_us":148,"threads_started":1}
I20260810 18:06:33.528784 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.524860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:60861 (local address 127.21.182.67:41805)
0810 18:06:33.525041 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:33.525046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:33.525288 (+ 242us) server_negotiation.cc:407] Connection header received
0810 18:06:33.525438 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:33.525440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:33.525504 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:33.525587 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:33.525903 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.526383 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.527661 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.528088 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.528228 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.528286 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:33.528587 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 18:06:33.528669 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 18:06:33.528815 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.524784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:32843 (local address 127.21.182.66:46533)
0810 18:06:33.524945 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:33.524952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:33.525212 (+ 260us) server_negotiation.cc:407] Connection header received
0810 18:06:33.525372 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:33.525378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:33.525446 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:33.525505 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:33.525868 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.526384 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.527626 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.528171 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.528293 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.528370 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:33.528615 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 18:06:33.528693 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 18:06:33.529172 27436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.524668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:46533 (local address 127.21.182.65:32843)
0810 18:06:33.525090 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:33.525103 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:33.525220 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:33.525546 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:33.525551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:33.525573 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:33.525740 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.525747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.526502 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.526507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.527495 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.527506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.528358 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.528361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.528473 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.528492 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:06:33.528951 (+ 459us) client_negotiation.cc:240] Negotiation successful
0810 18:06:33.529036 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":25,"thread_start_us":136,"threads_started":1}
I20260810 18:06:33.529546 27353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a21d19defb4245b005b56a2917ad67" candidate_uuid: "7c27b8e999b149ebbff718a5d598126c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56409990ef6421a98edd2e11a9f8279" is_pre_election: true
I20260810 18:06:33.529845 27204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a21d19defb4245b005b56a2917ad67" candidate_uuid: "7c27b8e999b149ebbff718a5d598126c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0464dff5be4a72b251132611e89e2a" is_pre_election: true
I20260810 18:06:33.529917 27353 raft_consensus.cc:2454] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c27b8e999b149ebbff718a5d598126c in term 0.
I20260810 18:06:33.529947 27434 leader_election.cc:288] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c27b8e999b149ebbff718a5d598126c (127.21.182.65:37983), a56409990ef6421a98edd2e11a9f8279 (127.21.182.67:41805)
I20260810 18:06:33.530035 27204 raft_consensus.cc:2454] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c27b8e999b149ebbff718a5d598126c in term 0.
I20260810 18:06:33.530457 26979 leader_election.cc:302] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c27b8e999b149ebbff718a5d598126c, a56409990ef6421a98edd2e11a9f8279; no voters:
I20260810 18:06:33.531080 27435 raft_consensus.cc:2781] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:33.531261 27435 raft_consensus.cc:482] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:33.531353 27435 raft_consensus.cc:3037] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:33.532553 27435 raft_consensus.cc:504] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c27b8e999b149ebbff718a5d598126c" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37983 } } peers { permanent_uuid: "2c0464dff5be4a72b251132611e89e2a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 46533 } } peers { permanent_uuid: "a56409990ef6421a98edd2e11a9f8279" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 41805 } }
I20260810 18:06:33.533200 27435 leader_election.cc:288] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [CANDIDATE]: Term 1 election: Requested vote from peers 2c0464dff5be4a72b251132611e89e2a (127.21.182.66:46533), a56409990ef6421a98edd2e11a9f8279 (127.21.182.67:41805)
I20260810 18:06:33.533291 27353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a21d19defb4245b005b56a2917ad67" candidate_uuid: "7c27b8e999b149ebbff718a5d598126c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56409990ef6421a98edd2e11a9f8279"
I20260810 18:06:33.533466 27353 raft_consensus.cc:3037] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:33.533514 27204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a21d19defb4245b005b56a2917ad67" candidate_uuid: "7c27b8e999b149ebbff718a5d598126c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0464dff5be4a72b251132611e89e2a"
I20260810 18:06:33.533620 27204 raft_consensus.cc:3037] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:33.534895 27204 raft_consensus.cc:2454] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c27b8e999b149ebbff718a5d598126c in term 1.
I20260810 18:06:33.534893 27353 raft_consensus.cc:2454] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c27b8e999b149ebbff718a5d598126c in term 1.
I20260810 18:06:33.535279 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.530410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:40193 (local address 127.21.182.65:37983)
0810 18:06:33.530704 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:33.530708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:33.530813 (+ 105us) server_negotiation.cc:407] Connection header received
0810 18:06:33.531024 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:33.531029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:33.531078 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:33.531150 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:33.531434 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.532223 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.534015 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.534593 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.534687 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.534735 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:33.535108 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 18:06:33.535180 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260810 18:06:33.535286 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.530229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:41805 (local address 127.21.182.66:44965)
0810 18:06:33.530954 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:33.530968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:33.531054 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:33.531349 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:33.531355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:33.531375 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:33.531474 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.531480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.532420 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.532422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.533868 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.533878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.534800 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.534803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.534940 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.534982 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 18:06:33.535059 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 18:06:33.535148 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":142,"thread_start_us":146,"threads_started":1}
I20260810 18:06:33.535354 26976 leader_election.cc:302] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c0464dff5be4a72b251132611e89e2a, 7c27b8e999b149ebbff718a5d598126c; no voters:
I20260810 18:06:33.535602 27435 raft_consensus.cc:2781] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:33.535630 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.530142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:37983 (local address 127.21.182.66:40193)
0810 18:06:33.530706 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:33.530723 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:33.530845 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:33.531167 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:33.531170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:33.531183 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:33.531307 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.531316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.532366 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.532372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.533867 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:33.533878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.534831 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.534834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:33.534942 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.534991 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 18:06:33.535407 (+ 416us) client_negotiation.cc:240] Negotiation successful
0810 18:06:33.535494 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":137}
I20260810 18:06:33.535838 27435 raft_consensus.cc:686] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [term 1 LEADER]: Becoming Leader. State: Replica: 7c27b8e999b149ebbff718a5d598126c, State: Running, Role: LEADER
I20260810 18:06:33.536235 27056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a21d19defb4245b005b56a2917ad67" candidate_uuid: "2c0464dff5be4a72b251132611e89e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c27b8e999b149ebbff718a5d598126c" is_pre_election: true
I20260810 18:06:33.536247 27435 consensus_queue.cc:237] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c27b8e999b149ebbff718a5d598126c" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37983 } } peers { permanent_uuid: "2c0464dff5be4a72b251132611e89e2a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 46533 } } peers { permanent_uuid: "a56409990ef6421a98edd2e11a9f8279" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 41805 } }
I20260810 18:06:33.536837 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.530832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:44965 (local address 127.21.182.67:41805)
0810 18:06:33.530934 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:33.530938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:33.531013 (+ 75us) server_negotiation.cc:407] Connection header received
0810 18:06:33.531133 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:33.531138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:33.531184 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:33.531247 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:33.531587 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.532295 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.534015 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.534593 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.534697 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.534735 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:33.535588 (+ 853us) server_negotiation.cc:299] Negotiation successful
0810 18:06:33.535669 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 18:06:33.537264 27353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a21d19defb4245b005b56a2917ad67" candidate_uuid: "2c0464dff5be4a72b251132611e89e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56409990ef6421a98edd2e11a9f8279" is_pre_election: true
I20260810 18:06:33.537439 27353 raft_consensus.cc:2379] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2c0464dff5be4a72b251132611e89e2a in current term 1: Already voted for candidate 7c27b8e999b149ebbff718a5d598126c in this term.
I20260810 18:06:33.537950 27131 leader_election.cc:302] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c0464dff5be4a72b251132611e89e2a; no voters: 7c27b8e999b149ebbff718a5d598126c, a56409990ef6421a98edd2e11a9f8279
I20260810 18:06:33.538275 27434 raft_consensus.cc:2726] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:06:33.540409 26882 catalog_manager.cc:5495] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c27b8e999b149ebbff718a5d598126c (127.21.182.65). New cstate: current_term: 1 leader_uuid: "7c27b8e999b149ebbff718a5d598126c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c27b8e999b149ebbff718a5d598126c" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c0464dff5be4a72b251132611e89e2a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 46533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a56409990ef6421a98edd2e11a9f8279" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 41805 } health_report { overall_health: UNKNOWN } } }
W20260810 18:06:33.577474 27409 tablet.cc:2369] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:06:33.583211 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.579738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46686 (local address 127.21.182.65:37983)
0810 18:06:33.579880 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:33.579884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:33.579916 (+ 32us) server_negotiation.cc:407] Connection header received
0810 18:06:33.580065 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:33.580068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:33.580127 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:33.580218 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:33.580479 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.580981 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.581791 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:33.582000 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:33.582075 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:33.582360 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:33.582820 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:33.582921 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:33.582974 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 18:06:33.583023 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:06:33.589870 27353 raft_consensus.cc:1261] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279 [term 1 FOLLOWER]: Refusing update from remote peer 7c27b8e999b149ebbff718a5d598126c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:33.589963 27204 raft_consensus.cc:1261] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [term 1 FOLLOWER]: Refusing update from remote peer 7c27b8e999b149ebbff718a5d598126c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 18:06:33.590773 27258 tablet.cc:2369] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:06:33.590747 27435 consensus_queue.cc:1035] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a56409990ef6421a98edd2e11a9f8279" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:33.590984 27439 consensus_queue.cc:1035] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c0464dff5be4a72b251132611e89e2a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 46533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:06:33.598001 27105 tablet.cc:2369] T 59a21d19defb4245b005b56a2917ad67 P 7c27b8e999b149ebbff718a5d598126c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:06:33.599777 27443 mvcc.cc:205] Tried to move back new op lower bound from 7317033752934789120 to 7317033752728268800. Current Snapshot: MvccSnapshot[applied={T|T < 7317033752934789120}]
I20260810 18:06:33.601599 27447 mvcc.cc:205] Tried to move back new op lower bound from 7317033752934789120 to 7317033752728268800. Current Snapshot: MvccSnapshot[applied={T|T < 7317033752934789120}]
I20260810 18:06:33.610122 26882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:33.610312 26882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:33.613310 26882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 393432dc26d2420aa8fc993ac98543a7: Sending DeleteTablet for 3 replicas of tablet 59a21d19defb4245b005b56a2917ad67
I20260810 18:06:33.614403 27032 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a21d19defb4245b005b56a2917ad67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:33 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:46684
I20260810 18:06:33.614404 27180 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a21d19defb4245b005b56a2917ad67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:33 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:53438
I20260810 18:06:33.615072 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 26959
I20260810 18:06:33.615818 27456 tablet_replica.cc:330] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a: stopping tablet replica
I20260810 18:06:33.615944 27330 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a21d19defb4245b005b56a2917ad67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:33 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:52818
I20260810 18:06:33.616294 27456 raft_consensus.cc:2229] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:33.616700 27456 raft_consensus.cc:2258] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:33.618285 27457 tablet_replica.cc:330] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279: stopping tablet replica
I20260810 18:06:33.618717 27457 raft_consensus.cc:2229] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:33.619105 27457 raft_consensus.cc:2258] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:33.620850 27457 ts_tablet_manager.cc:1905] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:33.622287 27456 ts_tablet_manager.cc:1905] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:33.623406 27457 ts_tablet_manager.cc:1918] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:33.623533 27457 log.cc:1196] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/59a21d19defb4245b005b56a2917ad67
I20260810 18:06:33.623983 27457 ts_tablet_manager.cc:1939] T 59a21d19defb4245b005b56a2917ad67 P a56409990ef6421a98edd2e11a9f8279: Deleting consensus metadata
I20260810 18:06:33.625329 26870 catalog_manager.cc:4841] TS a56409990ef6421a98edd2e11a9f8279 (127.21.182.67:41805): tablet 59a21d19defb4245b005b56a2917ad67 (table test-table [id=a78a32f0b9db4b899e2a8a05967595d3]) successfully deleted
I20260810 18:06:33.626292 27456 ts_tablet_manager.cc:1918] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:33.626423 27456 log.cc:1196] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/59a21d19defb4245b005b56a2917ad67
I20260810 18:06:33.627290 27456 ts_tablet_manager.cc:1939] T 59a21d19defb4245b005b56a2917ad67 P 2c0464dff5be4a72b251132611e89e2a: Deleting consensus metadata
W20260810 18:06:33.628867 26866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:37983 (error 108)
I20260810 18:06:33.629030 26868 catalog_manager.cc:4841] TS 2c0464dff5be4a72b251132611e89e2a (127.21.182.66:46533): tablet 59a21d19defb4245b005b56a2917ad67 (table test-table [id=a78a32f0b9db4b899e2a8a05967595d3]) successfully deleted
I20260810 18:06:33.629177 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 27109
I20260810 18:06:33.630317 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.630007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:37983 (local address 127.0.0.1:46696)
0810 18:06:33.630147 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:33.630227 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:37983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 18:06:33.630510 26866 catalog_manager.cc:4584] TS 7c27b8e999b149ebbff718a5d598126c (127.21.182.65:37983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59a21d19defb4245b005b56a2917ad67: Network error: Client connection negotiation failed: client connection to 127.21.182.65:37983: connect: Connection refused (error 111)
I20260810 18:06:33.642009 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 27262
I20260810 18:06:33.653354 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 26850
2026-08-10T18:06:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4126 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T18:06:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:33Z Disabled control of system clock
I20260810 18:06:33.690578 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:37873
--webserver_interface=127.21.182.126
--webserver_port=0
--builtin_ntp_servers=127.21.182.84:40359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:37873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 18:06:33.814189 27462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:33.814517 27462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:33.814558 27462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:33.818989 27462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:33.819073 27462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:33.819089 27462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:33.819108 27462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:33.819123 27462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:33.824294 27462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:40359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:37873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:37873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:33.825615 27462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:33.827292 27462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:06:33.837174 27462 server_base.cc:1034] running on GCE node
W20260810 18:06:33.837119 27468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:33.837108 27469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:33.837110 27471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:33.838184 27462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:33.839591 27462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:33.840778 27462 hybrid_clock.cc:648] HybridClock initialized: now 1786385193840763 us; error 31 us; skew 500 ppm
I20260810 18:06:33.843065 27462 webserver.cc:460] Webserver started at http://127.21.182.126:38605/ using document root <none> and password file <none>
I20260810 18:06:33.843747 27462 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:33.843824 27462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:33.844115 27462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:33.846526 27462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "4d7b5e7218ef42cebe2fd81ae7a6bc65"
format_stamp: "Formatted at 2026-08-10 18:06:33 on dist-test-slave-27sr"
server_key: "678e84355328ebf35a2c899b1a3df04c"
server_key_iv: "e56faedc0a9e21673273db8d82b65aa6"
server_key_version: "encryptionkey@0"
I20260810 18:06:33.847286 27462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "4d7b5e7218ef42cebe2fd81ae7a6bc65"
format_stamp: "Formatted at 2026-08-10 18:06:33 on dist-test-slave-27sr"
server_key: "678e84355328ebf35a2c899b1a3df04c"
server_key_iv: "e56faedc0a9e21673273db8d82b65aa6"
server_key_version: "encryptionkey@0"
I20260810 18:06:33.851784 27462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 18:06:33.854686 27477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:33.856276 27462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:33.856462 27462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "4d7b5e7218ef42cebe2fd81ae7a6bc65"
format_stamp: "Formatted at 2026-08-10 18:06:33 on dist-test-slave-27sr"
server_key: "678e84355328ebf35a2c899b1a3df04c"
server_key_iv: "e56faedc0a9e21673273db8d82b65aa6"
server_key_version: "encryptionkey@0"
I20260810 18:06:33.856611 27462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:33.861806 27462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:33.862653 27462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:33.862908 27462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:33.870678 27535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:37873 every 8 connection(s)
I20260810 18:06:33.870675 27462 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:37873
I20260810 18:06:33.872011 27462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:33.875607 27536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:33.877674 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 27462
I20260810 18:06:33.877825 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:33.878150 22233 external_mini_cluster.cc:1428] Setting key 4da4ae1f7902c1d97006a3b13017da66
I20260810 18:06:33.882483 27536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65: Bootstrap starting.
I20260810 18:06:33.885004 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:33.879794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33234 (local address 127.21.182.126:37873)
0810 18:06:33.880670 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:33.880696 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:33.880738 (+ 42us) server_negotiation.cc:407] Connection header received
0810 18:06:33.881652 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:33.881684 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:33.882120 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:33.882532 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:33.883428 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:33.883433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:33.883448 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:33.883460 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:33.883524 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:33.883542 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:33.883721 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:33.883822 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 18:06:33.884161 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":152,"threads_started":1}
I20260810 18:06:33.885270 27536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:33.886066 27536 log.cc:824] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:33.888293 27536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65: No bootstrap required, opened a new log
I20260810 18:06:33.892084 27536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7b5e7218ef42cebe2fd81ae7a6bc65" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37873 } }
I20260810 18:06:33.892380 27536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:33.892467 27536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7b5e7218ef42cebe2fd81ae7a6bc65, State: Initialized, Role: FOLLOWER
I20260810 18:06:33.892995 27536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7b5e7218ef42cebe2fd81ae7a6bc65" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37873 } }
I20260810 18:06:33.893175 27536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:33.893253 27536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:33.893388 27536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:33.894520 27536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7b5e7218ef42cebe2fd81ae7a6bc65" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37873 } }
I20260810 18:06:33.894940 27536 leader_election.cc:302] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d7b5e7218ef42cebe2fd81ae7a6bc65; no voters:
I20260810 18:06:33.895313 27536 leader_election.cc:288] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:33.895455 27541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:33.895740 27541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [term 1 LEADER]: Becoming Leader. State: Replica: 4d7b5e7218ef42cebe2fd81ae7a6bc65, State: Running, Role: LEADER
I20260810 18:06:33.896095 27541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7b5e7218ef42cebe2fd81ae7a6bc65" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37873 } }
I20260810 18:06:33.896750 27536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:33.897914 27542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d7b5e7218ef42cebe2fd81ae7a6bc65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7b5e7218ef42cebe2fd81ae7a6bc65" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37873 } } }
I20260810 18:06:33.898061 27542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:33.898361 27543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d7b5e7218ef42cebe2fd81ae7a6bc65. Latest consensus state: current_term: 1 leader_uuid: "4d7b5e7218ef42cebe2fd81ae7a6bc65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d7b5e7218ef42cebe2fd81ae7a6bc65" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37873 } } }
I20260810 18:06:33.898458 27543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:33.899631 27550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:33.902941 27550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:33.908942 27550 catalog_manager.cc:1347] Generated new cluster ID: 19e4bf2ec6214f08989a55a221c3fff8
I20260810 18:06:33.909039 27550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:33.922971 27550 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:33.923391 27550 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:33.935071 27550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65: Generated new TSK 0
I20260810 18:06:33.936139 27550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:06:33.942407 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37873
--builtin_ntp_servers=127.21.182.84:40359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 18:06:34.065996 27560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:34.066341 27560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:34.066419 27560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:34.070760 27560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:34.070863 27560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:34.070963 27560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:34.076217 27560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:40359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:34.077518 27560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:34.079165 27560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:34.089833 27565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:34.089882 27568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:34.090095 27560 server_base.cc:1034] running on GCE node
W20260810 18:06:34.090232 27566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:34.090734 27560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:34.091500 27560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:34.092700 27560 hybrid_clock.cc:648] HybridClock initialized: now 1786385194092679 us; error 33 us; skew 500 ppm
I20260810 18:06:34.094789 27560 webserver.cc:460] Webserver started at http://127.21.182.65:36619/ using document root <none> and password file <none>
I20260810 18:06:34.095450 27560 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:34.095553 27560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:34.095816 27560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:34.098448 27560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "d9355d32154e47109a31a2f93c8b01a3"
format_stamp: "Formatted at 2026-08-10 18:06:34 on dist-test-slave-27sr"
server_key: "5a00bfe01910323dc46f303ca353874c"
server_key_iv: "6fedcb8e361fe6c0fea682361457c398"
server_key_version: "encryptionkey@0"
I20260810 18:06:34.099200 27560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "d9355d32154e47109a31a2f93c8b01a3"
format_stamp: "Formatted at 2026-08-10 18:06:34 on dist-test-slave-27sr"
server_key: "5a00bfe01910323dc46f303ca353874c"
server_key_iv: "6fedcb8e361fe6c0fea682361457c398"
server_key_version: "encryptionkey@0"
I20260810 18:06:34.103489 27560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:06:34.106599 27574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:34.108109 27560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:34.108259 27560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "d9355d32154e47109a31a2f93c8b01a3"
format_stamp: "Formatted at 2026-08-10 18:06:34 on dist-test-slave-27sr"
server_key: "5a00bfe01910323dc46f303ca353874c"
server_key_iv: "6fedcb8e361fe6c0fea682361457c398"
server_key_version: "encryptionkey@0"
I20260810 18:06:34.108407 27560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:34.113633 27560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:34.114374 27560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:34.114598 27560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:34.115199 27560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:34.116277 27560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:34.116350 27560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:34.116436 27560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:34.116487 27560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:34.125487 27560 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:36421
I20260810 18:06:34.125553 27699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:36421 every 8 connection(s)
I20260810 18:06:34.126637 27560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:34.130764 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 27560
I20260810 18:06:34.130927 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:06:34.131240 22233 external_mini_cluster.cc:1428] Setting key 702a95ca333a1817ee451a168979ad66
I20260810 18:06:34.133421 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37873
--builtin_ntp_servers=127.21.182.84:40359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 18:06:34.133709 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.128863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:36311 (local address 127.21.182.126:37873)
0810 18:06:34.129012 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:34.129016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:34.130275 (+ 1259us) server_negotiation.cc:407] Connection header received
0810 18:06:34.131813 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:34.131816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:34.131869 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:34.131986 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:34.133208 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:34.133212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:34.133215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:34.133228 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:34.133254 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:34.133269 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:34.133360 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:34.133522 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 18:06:34.133571 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:06:34.134639 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.129306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37873 (local address 127.21.182.65:36311)
0810 18:06:34.130096 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:34.130144 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:34.131574 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:34.132179 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:34.132187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:34.132283 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:34.132928 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:06:34.132930 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:34.132975 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:06:34.133052 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:34.133360 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:34.133370 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:34.133378 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 18:06:34.133573 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 18:06:34.133842 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":124,"threads_started":1}
I20260810 18:06:34.136267 27700 heartbeater.cc:347] Connected to a master server at 127.21.182.126:37873
I20260810 18:06:34.136528 27700 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:34.136996 27700 heartbeater.cc:511] Master 127.21.182.126:37873 requested a full tablet report, sending...
I20260810 18:06:34.138845 27495 ts_manager.cc:194] Registered new tserver with Master: d9355d32154e47109a31a2f93c8b01a3 (127.21.182.65:36421)
W20260810 18:06:34.256662 27703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:34.256981 27703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:34.257028 27703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:34.261281 27703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:34.261370 27703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:34.261433 27703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:34.266492 27703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:40359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:34.267724 27703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:34.269340 27703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:06:34.280157 27703 server_base.cc:1034] running on GCE node
W20260810 18:06:34.280102 27709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:34.280102 27710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:34.280104 27712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:34.280882 27703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:34.281567 27703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:34.282745 27703 hybrid_clock.cc:648] HybridClock initialized: now 1786385194282721 us; error 35 us; skew 500 ppm
I20260810 18:06:34.284910 27703 webserver.cc:460] Webserver started at http://127.21.182.66:32807/ using document root <none> and password file <none>
I20260810 18:06:34.285526 27703 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:34.285650 27703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:34.285933 27703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:34.288432 27703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "96b15c4d35a54d37867a0a9750adc41a"
format_stamp: "Formatted at 2026-08-10 18:06:34 on dist-test-slave-27sr"
server_key: "1d11adf67f941808464ad1aa70b151d5"
server_key_iv: "de29a62426eece01737a8850e05f81ac"
server_key_version: "encryptionkey@0"
I20260810 18:06:34.289052 27703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "96b15c4d35a54d37867a0a9750adc41a"
format_stamp: "Formatted at 2026-08-10 18:06:34 on dist-test-slave-27sr"
server_key: "1d11adf67f941808464ad1aa70b151d5"
server_key_iv: "de29a62426eece01737a8850e05f81ac"
server_key_version: "encryptionkey@0"
I20260810 18:06:34.293283 27703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:06:34.296125 27718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:34.297577 27703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:34.297734 27703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "96b15c4d35a54d37867a0a9750adc41a"
format_stamp: "Formatted at 2026-08-10 18:06:34 on dist-test-slave-27sr"
server_key: "1d11adf67f941808464ad1aa70b151d5"
server_key_iv: "de29a62426eece01737a8850e05f81ac"
server_key_version: "encryptionkey@0"
I20260810 18:06:34.297894 27703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:34.303057 27703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:34.303802 27703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:34.304064 27703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:34.304675 27703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:34.305701 27703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:34.305773 27703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:34.305848 27703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:34.305883 27703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:34.315168 27703 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:40375
I20260810 18:06:34.315222 27846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:40375 every 8 connection(s)
I20260810 18:06:34.316301 27703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:06:34.322947 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.318396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:45967 (local address 127.21.182.126:37873)
0810 18:06:34.318563 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:34.318569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:34.319754 (+ 1185us) server_negotiation.cc:407] Connection header received
0810 18:06:34.321111 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:34.321114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:34.321155 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:34.321237 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:34.322487 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:34.322490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:34.322492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:34.322505 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:34.322524 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:34.322532 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:34.322633 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:34.322774 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 18:06:34.322817 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 18:06:34.323554 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 27703
I20260810 18:06:34.323660 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:06:34.324018 22233 external_mini_cluster.cc:1428] Setting key 373b87dc55be32226c60fb805a9b7bff
I20260810 18:06:34.324102 27849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.318891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37873 (local address 127.21.182.66:45967)
0810 18:06:34.319588 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:34.319629 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:34.320895 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:34.321466 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:34.321474 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:34.321579 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:34.322202 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:06:34.322204 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:34.322247 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:06:34.322330 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:34.322640 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:34.322647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:34.322652 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 18:06:34.322794 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 18:06:34.323036 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":118,"threads_started":1}
I20260810 18:06:34.325423 27847 heartbeater.cc:347] Connected to a master server at 127.21.182.126:37873
I20260810 18:06:34.325711 27847 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:34.326228 27847 heartbeater.cc:511] Master 127.21.182.126:37873 requested a full tablet report, sending...
I20260810 18:06:34.326562 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37873
--builtin_ntp_servers=127.21.182.84:40359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 18:06:34.327527 27494 ts_manager.cc:194] Registered new tserver with Master: 96b15c4d35a54d37867a0a9750adc41a (127.21.182.66:40375)
W20260810 18:06:34.451601 27850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:34.452003 27850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:34.452087 27850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:34.456461 27850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:34.456529 27850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:34.456627 27850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:34.461947 27850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:40359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:34.463225 27850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:34.464926 27850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:34.475986 27855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:34.476001 27857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:34.476410 27850 server_base.cc:1034] running on GCE node
W20260810 18:06:34.476049 27859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:34.476933 27850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:34.477667 27850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:34.478883 27850 hybrid_clock.cc:648] HybridClock initialized: now 1786385194478852 us; error 50 us; skew 500 ppm
I20260810 18:06:34.480981 27850 webserver.cc:460] Webserver started at http://127.21.182.67:40129/ using document root <none> and password file <none>
I20260810 18:06:34.481639 27850 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:34.481709 27850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:34.482033 27850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:34.484462 27850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "2ba1c35db04a439c910159bed1dbae5c"
format_stamp: "Formatted at 2026-08-10 18:06:34 on dist-test-slave-27sr"
server_key: "9292e5a03d091ae84aec2bbde3bdf1e5"
server_key_iv: "69a24ce6c7cca574cf295179700d5a7e"
server_key_version: "encryptionkey@0"
I20260810 18:06:34.485198 27850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "2ba1c35db04a439c910159bed1dbae5c"
format_stamp: "Formatted at 2026-08-10 18:06:34 on dist-test-slave-27sr"
server_key: "9292e5a03d091ae84aec2bbde3bdf1e5"
server_key_iv: "69a24ce6c7cca574cf295179700d5a7e"
server_key_version: "encryptionkey@0"
I20260810 18:06:34.489575 27850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:06:34.492667 27867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:34.494163 27850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:34.494364 27850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "2ba1c35db04a439c910159bed1dbae5c"
format_stamp: "Formatted at 2026-08-10 18:06:34 on dist-test-slave-27sr"
server_key: "9292e5a03d091ae84aec2bbde3bdf1e5"
server_key_iv: "69a24ce6c7cca574cf295179700d5a7e"
server_key_version: "encryptionkey@0"
I20260810 18:06:34.494513 27850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:34.500025 27850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:34.500742 27850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:34.500955 27850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:34.501674 27850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:34.502959 27850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:34.503034 27850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:34.503115 27850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:34.503157 27850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:34.512768 27850 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:40237
I20260810 18:06:34.512825 28001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:40237 every 8 connection(s)
I20260810 18:06:34.513880 27850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:06:34.515751 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 27850
I20260810 18:06:34.515934 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:06:34.516256 22233 external_mini_cluster.cc:1428] Setting key b8b8cf8a172330c260c60197c997dbcf
I20260810 18:06:34.521128 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.515967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:33945 (local address 127.21.182.126:37873)
0810 18:06:34.516681 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:34.516687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:34.517667 (+ 980us) server_negotiation.cc:407] Connection header received
0810 18:06:34.519322 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:34.519326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:34.519371 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:34.519452 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:34.520616 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:34.520620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:34.520623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:34.520648 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:34.520672 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:34.520680 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:34.520782 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:34.520936 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 18:06:34.520985 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600}
I20260810 18:06:34.522071 28005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.516545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37873 (local address 127.21.182.67:33945)
0810 18:06:34.517514 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:34.517576 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:34.519089 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:34.519632 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:34.519640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:34.519734 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:34.520336 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:06:34.520338 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:34.520381 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:06:34.520457 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:34.520775 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:34.520784 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:34.520792 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 18:06:34.520986 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 18:06:34.521247 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":149,"threads_started":1}
I20260810 18:06:34.523295 28003 heartbeater.cc:347] Connected to a master server at 127.21.182.126:37873
I20260810 18:06:34.523532 28003 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:34.523955 28003 heartbeater.cc:511] Master 127.21.182.126:37873 requested a full tablet report, sending...
I20260810 18:06:34.525002 27494 ts_manager.cc:194] Registered new tserver with Master: 2ba1c35db04a439c910159bed1dbae5c (127.21.182.67:40237)
I20260810 18:06:34.530563 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:06:34.534413 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.532595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33246 (local address 127.21.182.126:37873)
0810 18:06:34.532753 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:34.532758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:34.532949 (+ 191us) server_negotiation.cc:407] Connection header received
0810 18:06:34.533117 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:34.533120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:34.533161 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:34.533254 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:34.533961 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:34.533964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:34.533967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:34.533980 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:34.534002 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:34.534017 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:34.534109 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:34.534204 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 18:06:34.534250 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:06:34.537384 27494 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:06:34.540701 27494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:34.553978 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.550108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:36421 (local address 127.0.0.1:41870)
0810 18:06:34.550738 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:34.550792 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:34.550909 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:34.552390 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:34.552395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:34.552458 (+ 63us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:34.553062 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:06:34.553064 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:34.553090 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:06:34.553170 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:34.553586 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:34.553595 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:34.553601 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:06:34.553729 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 18:06:34.553809 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":96,"threads_started":1}
I20260810 18:06:34.554033 28017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.550232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.21.182.65:36421)
0810 18:06:34.551208 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:34.551220 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:34.551244 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:06:34.551298 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:34.551302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:34.551522 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:34.551669 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:34.553370 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:34.553378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:34.553392 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:34.553404 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:34.553466 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:34.553509 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:34.553621 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:34.553755 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 18:06:34.553853 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":115,"threads_started":1}
I20260810 18:06:34.554270 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.550980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.21.182.67:40237)
0810 18:06:34.551598 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:34.551610 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:34.551632 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:06:34.551684 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:34.551690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:34.551873 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:34.552041 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:34.553571 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:34.553579 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:34.553591 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:34.553603 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:34.553649 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:34.553689 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:34.553795 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:34.554045 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 18:06:34.554128 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":155,"threads_started":1}
I20260810 18:06:34.554392 28016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.550178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:40237 (local address 127.0.0.1:42090)
0810 18:06:34.551033 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:34.551048 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:34.551121 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:34.552686 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:34.552689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:34.552705 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:34.553364 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:06:34.553366 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:34.553378 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:06:34.553406 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:34.553843 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:34.553846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:34.553849 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 18:06:34.553923 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 18:06:34.553956 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":125,"threads_started":1}
I20260810 18:06:34.554828 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.550973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.21.182.66:40375)
0810 18:06:34.551695 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:34.551705 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:34.551724 (+ 19us) server_negotiation.cc:407] Connection header received
0810 18:06:34.551788 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:34.551792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:34.551951 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:34.552069 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:34.553546 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:34.553553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:34.553570 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:34.553584 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:34.553632 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:34.553670 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:34.553773 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:34.554590 (+ 817us) server_negotiation.cc:299] Negotiation successful
0810 18:06:34.554660 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":107,"threads_started":1}
I20260810 18:06:34.554888 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.550494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:40375 (local address 127.0.0.1:35330)
0810 18:06:34.551353 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:34.551369 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:34.551455 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:34.552686 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:34.552689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:34.552705 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:34.553385 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:06:34.553388 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:34.553397 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:06:34.553418 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:34.553815 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:34.553818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:34.553821 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 18:06:34.553899 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 18:06:34.553927 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":106,"threads_started":1}
I20260810 18:06:34.557476 27627 tablet_service.cc:1467] Processing CreateTablet for tablet 66eaaad4ebd04cc7aeff1faaff137c3e (DEFAULT_TABLE table=test-table [id=bd290a598b1445cc9bd0a11248e1c6b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:34.557508 27769 tablet_service.cc:1467] Processing CreateTablet for tablet 66eaaad4ebd04cc7aeff1faaff137c3e (DEFAULT_TABLE table=test-table [id=bd290a598b1445cc9bd0a11248e1c6b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:34.557508 27925 tablet_service.cc:1467] Processing CreateTablet for tablet 66eaaad4ebd04cc7aeff1faaff137c3e (DEFAULT_TABLE table=test-table [id=bd290a598b1445cc9bd0a11248e1c6b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:34.558712 27627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66eaaad4ebd04cc7aeff1faaff137c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:34.558723 27925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66eaaad4ebd04cc7aeff1faaff137c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:34.558723 27769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66eaaad4ebd04cc7aeff1faaff137c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:34.565889 28021 tablet_bootstrap.cc:492] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3: Bootstrap starting.
I20260810 18:06:34.567517 28022 tablet_bootstrap.cc:492] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a: Bootstrap starting.
I20260810 18:06:34.567518 28023 tablet_bootstrap.cc:492] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c: Bootstrap starting.
I20260810 18:06:34.569088 28021 tablet_bootstrap.cc:654] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:34.569584 28023 tablet_bootstrap.cc:654] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:34.569813 28022 tablet_bootstrap.cc:654] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:34.570081 28021 log.cc:824] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:34.570417 28023 log.cc:824] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:34.570642 28022 log.cc:824] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:34.572549 28021 tablet_bootstrap.cc:492] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3: No bootstrap required, opened a new log
I20260810 18:06:34.572640 28023 tablet_bootstrap.cc:492] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c: No bootstrap required, opened a new log
I20260810 18:06:34.572779 28021 ts_tablet_manager.cc:1397] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:06:34.572841 28023 ts_tablet_manager.cc:1397] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260810 18:06:34.573194 28022 tablet_bootstrap.cc:492] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a: No bootstrap required, opened a new log
I20260810 18:06:34.573379 28022 ts_tablet_manager.cc:1397] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:06:34.576651 28021 raft_consensus.cc:348] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9355d32154e47109a31a2f93c8b01a3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 36421 } } peers { permanent_uuid: "2ba1c35db04a439c910159bed1dbae5c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40237 } } peers { permanent_uuid: "96b15c4d35a54d37867a0a9750adc41a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40375 } }
I20260810 18:06:34.576925 28021 raft_consensus.cc:374] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:34.576984 28021 raft_consensus.cc:729] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9355d32154e47109a31a2f93c8b01a3, State: Initialized, Role: FOLLOWER
I20260810 18:06:34.577471 28021 consensus_queue.cc:260] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9355d32154e47109a31a2f93c8b01a3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 36421 } } peers { permanent_uuid: "2ba1c35db04a439c910159bed1dbae5c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40237 } } peers { permanent_uuid: "96b15c4d35a54d37867a0a9750adc41a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40375 } }
I20260810 18:06:34.577551 28023 raft_consensus.cc:348] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9355d32154e47109a31a2f93c8b01a3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 36421 } } peers { permanent_uuid: "2ba1c35db04a439c910159bed1dbae5c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40237 } } peers { permanent_uuid: "96b15c4d35a54d37867a0a9750adc41a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40375 } }
I20260810 18:06:34.577838 28023 raft_consensus.cc:374] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:34.577930 28023 raft_consensus.cc:729] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba1c35db04a439c910159bed1dbae5c, State: Initialized, Role: FOLLOWER
I20260810 18:06:34.577941 28022 raft_consensus.cc:348] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9355d32154e47109a31a2f93c8b01a3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 36421 } } peers { permanent_uuid: "2ba1c35db04a439c910159bed1dbae5c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40237 } } peers { permanent_uuid: "96b15c4d35a54d37867a0a9750adc41a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40375 } }
I20260810 18:06:34.578183 28022 raft_consensus.cc:374] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:34.578238 28022 raft_consensus.cc:729] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b15c4d35a54d37867a0a9750adc41a, State: Initialized, Role: FOLLOWER
I20260810 18:06:34.578378 28023 consensus_queue.cc:260] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9355d32154e47109a31a2f93c8b01a3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 36421 } } peers { permanent_uuid: "2ba1c35db04a439c910159bed1dbae5c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40237 } } peers { permanent_uuid: "96b15c4d35a54d37867a0a9750adc41a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40375 } }
I20260810 18:06:34.578704 28021 ts_tablet_manager.cc:1428] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:06:34.578630 28022 consensus_queue.cc:260] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9355d32154e47109a31a2f93c8b01a3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 36421 } } peers { permanent_uuid: "2ba1c35db04a439c910159bed1dbae5c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40237 } } peers { permanent_uuid: "96b15c4d35a54d37867a0a9750adc41a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40375 } }
I20260810 18:06:34.579128 27700 heartbeater.cc:503] Master 127.21.182.126:37873 was elected leader, sending a full tablet report...
I20260810 18:06:34.579406 28023 ts_tablet_manager.cc:1428] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260810 18:06:34.579515 28022 ts_tablet_manager.cc:1428] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:06:34.579856 27847 heartbeater.cc:503] Master 127.21.182.126:37873 was elected leader, sending a full tablet report...
I20260810 18:06:34.581149 28003 heartbeater.cc:503] Master 127.21.182.126:37873 was elected leader, sending a full tablet report...
W20260810 18:06:34.628051 27701 tablet.cc:2369] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:06:34.703446 28030 raft_consensus.cc:482] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:34.703701 28030 raft_consensus.cc:504] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9355d32154e47109a31a2f93c8b01a3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 36421 } } peers { permanent_uuid: "2ba1c35db04a439c910159bed1dbae5c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40237 } } peers { permanent_uuid: "96b15c4d35a54d37867a0a9750adc41a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40375 } }
I20260810 18:06:34.705156 28030 leader_election.cc:288] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9355d32154e47109a31a2f93c8b01a3 (127.21.182.65:36421), 96b15c4d35a54d37867a0a9750adc41a (127.21.182.66:40375)
I20260810 18:06:34.707270 28017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.705526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:53643 (local address 127.21.182.65:36421)
0810 18:06:34.705659 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:34.705665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:34.705680 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:06:34.705838 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:34.705842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:34.705898 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:34.705986 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:34.706804 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:34.706810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:34.706817 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:34.706830 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:34.706850 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:34.706860 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:34.707011 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:34.707060 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 18:06:34.707110 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:06:34.707280 28005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.705404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:36421 (local address 127.21.182.67:53643)
0810 18:06:34.705609 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:34.705626 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:34.705715 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:34.706035 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:34.706038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:34.706054 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:34.706634 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:06:34.706635 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:34.706645 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:06:34.706672 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:34.706976 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:34.706981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:34.706984 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 18:06:34.707060 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 18:06:34.707097 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260810 18:06:34.707434 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.705637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:38013 (local address 127.21.182.66:40375)
0810 18:06:34.705822 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:34.705826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:34.705982 (+ 156us) server_negotiation.cc:407] Connection header received
0810 18:06:34.706063 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:34.706066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:34.706111 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:34.706205 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:34.707018 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:34.707022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:34.707024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:34.707043 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:34.707067 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:34.707077 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:34.707171 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:34.707256 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 18:06:34.707298 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:06:34.707710 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.705460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:40375 (local address 127.21.182.67:38013)
0810 18:06:34.705848 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:34.705872 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:34.705975 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:34.706260 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:34.706264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:34.706282 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:34.706850 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:06:34.706851 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:34.706859 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:06:34.706879 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:34.707173 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:34.707176 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:34.707179 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 18:06:34.707256 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 18:06:34.707291 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":138,"threads_started":1}
I20260810 18:06:34.708333 27647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66eaaad4ebd04cc7aeff1faaff137c3e" candidate_uuid: "2ba1c35db04a439c910159bed1dbae5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9355d32154e47109a31a2f93c8b01a3" is_pre_election: true
I20260810 18:06:34.708411 27795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66eaaad4ebd04cc7aeff1faaff137c3e" candidate_uuid: "2ba1c35db04a439c910159bed1dbae5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b15c4d35a54d37867a0a9750adc41a" is_pre_election: true
I20260810 18:06:34.708726 27795 raft_consensus.cc:2454] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ba1c35db04a439c910159bed1dbae5c in term 0.
I20260810 18:06:34.708726 27647 raft_consensus.cc:2454] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ba1c35db04a439c910159bed1dbae5c in term 0.
I20260810 18:06:34.709342 27869 leader_election.cc:302] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba1c35db04a439c910159bed1dbae5c, 96b15c4d35a54d37867a0a9750adc41a; no voters:
I20260810 18:06:34.709707 28030 raft_consensus.cc:2781] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:34.709797 28030 raft_consensus.cc:482] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:34.709895 28030 raft_consensus.cc:3037] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:34.711037 28030 raft_consensus.cc:504] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9355d32154e47109a31a2f93c8b01a3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 36421 } } peers { permanent_uuid: "2ba1c35db04a439c910159bed1dbae5c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40237 } } peers { permanent_uuid: "96b15c4d35a54d37867a0a9750adc41a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40375 } }
I20260810 18:06:34.711464 28030 leader_election.cc:288] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [CANDIDATE]: Term 1 election: Requested vote from peers d9355d32154e47109a31a2f93c8b01a3 (127.21.182.65:36421), 96b15c4d35a54d37867a0a9750adc41a (127.21.182.66:40375)
I20260810 18:06:34.712009 27647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66eaaad4ebd04cc7aeff1faaff137c3e" candidate_uuid: "2ba1c35db04a439c910159bed1dbae5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9355d32154e47109a31a2f93c8b01a3"
I20260810 18:06:34.712163 27647 raft_consensus.cc:3037] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:34.712277 27795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66eaaad4ebd04cc7aeff1faaff137c3e" candidate_uuid: "2ba1c35db04a439c910159bed1dbae5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b15c4d35a54d37867a0a9750adc41a"
I20260810 18:06:34.712473 27795 raft_consensus.cc:3037] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:34.713575 27647 raft_consensus.cc:2454] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ba1c35db04a439c910159bed1dbae5c in term 1.
I20260810 18:06:34.713917 27868 leader_election.cc:302] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba1c35db04a439c910159bed1dbae5c, d9355d32154e47109a31a2f93c8b01a3; no voters:
I20260810 18:06:34.714010 27795 raft_consensus.cc:2454] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ba1c35db04a439c910159bed1dbae5c in term 1.
I20260810 18:06:34.714123 28030 raft_consensus.cc:2781] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:34.714376 28030 raft_consensus.cc:686] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [term 1 LEADER]: Becoming Leader. State: Replica: 2ba1c35db04a439c910159bed1dbae5c, State: Running, Role: LEADER
I20260810 18:06:34.714746 28030 consensus_queue.cc:237] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9355d32154e47109a31a2f93c8b01a3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 36421 } } peers { permanent_uuid: "2ba1c35db04a439c910159bed1dbae5c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40237 } } peers { permanent_uuid: "96b15c4d35a54d37867a0a9750adc41a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40375 } }
I20260810 18:06:34.718227 27495 catalog_manager.cc:5495] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ba1c35db04a439c910159bed1dbae5c (127.21.182.67). New cstate: current_term: 1 leader_uuid: "2ba1c35db04a439c910159bed1dbae5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9355d32154e47109a31a2f93c8b01a3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ba1c35db04a439c910159bed1dbae5c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96b15c4d35a54d37867a0a9750adc41a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40375 } health_report { overall_health: UNKNOWN } } }
I20260810 18:06:34.748360 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.746437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.21.182.67:40237)
0810 18:06:34.746605 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:34.746610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:34.746628 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:06:34.746872 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:34.746875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:34.746956 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:34.747064 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:34.747844 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:34.747853 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:34.747855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:34.747867 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:34.747940 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:34.747956 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:34.748059 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:34.748156 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 18:06:34.748207 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:06:34.755126 27795 raft_consensus.cc:1261] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a [term 1 FOLLOWER]: Refusing update from remote peer 2ba1c35db04a439c910159bed1dbae5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:34.755126 27647 raft_consensus.cc:1261] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3 [term 1 FOLLOWER]: Refusing update from remote peer 2ba1c35db04a439c910159bed1dbae5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:34.756064 28034 consensus_queue.cc:1035] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9355d32154e47109a31a2f93c8b01a3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:34.756278 28030 consensus_queue.cc:1035] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b15c4d35a54d37867a0a9750adc41a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:34.764017 28037 mvcc.cc:205] Tried to move back new op lower bound from 7317033757708455936 to 7317033757554442240. Current Snapshot: MvccSnapshot[applied={T|T < 7317033757708455936}]
I20260810 18:06:34.764290 28039 mvcc.cc:205] Tried to move back new op lower bound from 7317033757708455936 to 7317033757554442240. Current Snapshot: MvccSnapshot[applied={T|T < 7317033757708455936}]
I20260810 18:06:34.774072 27495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:34.774248 27495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:34.777107 27495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d7b5e7218ef42cebe2fd81ae7a6bc65: Sending DeleteTablet for 3 replicas of tablet 66eaaad4ebd04cc7aeff1faaff137c3e
I20260810 18:06:34.778044 27925 tablet_service.cc:1514] Processing DeleteTablet for tablet 66eaaad4ebd04cc7aeff1faaff137c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:34 UTC) from {username='slave'} at 127.0.0.1:42090
I20260810 18:06:34.778045 27769 tablet_service.cc:1514] Processing DeleteTablet for tablet 66eaaad4ebd04cc7aeff1faaff137c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:34 UTC) from {username='slave'} at 127.0.0.1:35330
I20260810 18:06:34.778103 27627 tablet_service.cc:1514] Processing DeleteTablet for tablet 66eaaad4ebd04cc7aeff1faaff137c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:34 UTC) from {username='slave'} at 127.0.0.1:41870
I20260810 18:06:34.778717 28050 tablet_replica.cc:330] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3: stopping tablet replica
I20260810 18:06:34.779114 28050 raft_consensus.cc:2229] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:34.779546 28050 raft_consensus.cc:2258] T 66eaaad4ebd04cc7aeff1faaff137c3e P d9355d32154e47109a31a2f93c8b01a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:34.779742 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 27560
I20260810 18:06:34.780020 28048 tablet_replica.cc:330] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a: stopping tablet replica
I20260810 18:06:34.780349 28048 raft_consensus.cc:2229] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:34.780648 28048 raft_consensus.cc:2258] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:34.781450 28049 tablet_replica.cc:330] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c: stopping tablet replica
I20260810 18:06:34.781736 28049 raft_consensus.cc:2229] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:06:34.782140 28049 raft_consensus.cc:2258] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:34.783222 28048 ts_tablet_manager.cc:1905] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:34.787364 28048 ts_tablet_manager.cc:1918] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:34.787511 28048 log.cc:1196] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/66eaaad4ebd04cc7aeff1faaff137c3e
I20260810 18:06:34.787945 28048 ts_tablet_manager.cc:1939] T 66eaaad4ebd04cc7aeff1faaff137c3e P 96b15c4d35a54d37867a0a9750adc41a: Deleting consensus metadata
I20260810 18:06:34.789574 27479 catalog_manager.cc:4841] TS 96b15c4d35a54d37867a0a9750adc41a (127.21.182.66:40375): tablet 66eaaad4ebd04cc7aeff1faaff137c3e (table test-table [id=bd290a598b1445cc9bd0a11248e1c6b6]) successfully deleted
I20260810 18:06:34.790663 28049 ts_tablet_manager.cc:1905] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 18:06:34.791874 27478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:36421 (error 108)
I20260810 18:06:34.792882 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 27703
I20260810 18:06:34.793155 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:34.792730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:36421 (local address 127.0.0.1:41876)
0810 18:06:34.792955 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:34.793041 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260810 18:06:34.793494 27478 catalog_manager.cc:4584] TS d9355d32154e47109a31a2f93c8b01a3 (127.21.182.65:36421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66eaaad4ebd04cc7aeff1faaff137c3e: Network error: Client connection negotiation failed: client connection to 127.21.182.65:36421: connect: Connection refused (error 111)
I20260810 18:06:34.794049 28049 ts_tablet_manager.cc:1918] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:34.794142 28049 log.cc:1196] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/66eaaad4ebd04cc7aeff1faaff137c3e
I20260810 18:06:34.794510 28049 ts_tablet_manager.cc:1939] T 66eaaad4ebd04cc7aeff1faaff137c3e P 2ba1c35db04a439c910159bed1dbae5c: Deleting consensus metadata
I20260810 18:06:34.797061 27481 catalog_manager.cc:4841] TS 2ba1c35db04a439c910159bed1dbae5c (127.21.182.67:40237): tablet 66eaaad4ebd04cc7aeff1faaff137c3e (table test-table [id=bd290a598b1445cc9bd0a11248e1c6b6]) successfully deleted
I20260810 18:06:34.804756 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 27850
I20260810 18:06:34.814664 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 27462
2026-08-10T18:06:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1161 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 18:06:34.842806 22233 mini_oidc.cc:152] Starting JWKS server
I20260810 18:06:34.844739 22233 webserver.cc:460] Webserver started at https://127.0.0.1:41723/ using document root <none> and password file <none>
I20260810 18:06:34.844893 22233 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 18:06:34.845170 22233 webserver.cc:460] Webserver started at http://127.0.0.1:41175/ using document root <none> and password file <none>
2026-08-10T18:06:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:34Z Disabled control of system clock
I20260810 18:06:34.856019 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:34631
--webserver_interface=127.21.182.126
--webserver_port=0
--builtin_ntp_servers=127.21.182.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:34631
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41723/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/testchainca.cert with env {}
W20260810 18:06:34.977926 28059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:34.978216 28059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:34.978256 28059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:34.982303 28059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:34.982370 28059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:34.982386 28059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:34.982403 28059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:34.982419 28059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:06:34.982435 28059 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:06:34.982448 28059 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41723/jwks.json
I20260810 18:06:34.988014 28059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:34631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:34631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41723/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:34.989146 28059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:34.990226 28059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:34.999183 28064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:34.999289 28065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:34.999382 28059 server_base.cc:1034] running on GCE node
W20260810 18:06:34.999183 28068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:35.000269 28059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:35.001425 28059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:35.002592 28059 hybrid_clock.cc:648] HybridClock initialized: now 1786385195002567 us; error 37 us; skew 500 ppm
I20260810 18:06:35.004639 28059 webserver.cc:460] Webserver started at http://127.21.182.126:34233/ using document root <none> and password file <none>
I20260810 18:06:35.005245 28059 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:35.005316 28059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:35.005568 28059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:35.007743 28059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "b8435fdb5f994cefbab6a2f1c2a36565"
format_stamp: "Formatted at 2026-08-10 18:06:35 on dist-test-slave-27sr"
server_key: "44edbf91111bf25eb3d10c7cda365151"
server_key_iv: "3d2c4e7ef2a4657d71970c3cf9a84c6e"
server_key_version: "encryptionkey@0"
I20260810 18:06:35.008438 28059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "b8435fdb5f994cefbab6a2f1c2a36565"
format_stamp: "Formatted at 2026-08-10 18:06:35 on dist-test-slave-27sr"
server_key: "44edbf91111bf25eb3d10c7cda365151"
server_key_iv: "3d2c4e7ef2a4657d71970c3cf9a84c6e"
server_key_version: "encryptionkey@0"
I20260810 18:06:35.012475 28059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:06:35.015045 28074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:35.016463 28059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:35.016618 28059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "b8435fdb5f994cefbab6a2f1c2a36565"
format_stamp: "Formatted at 2026-08-10 18:06:35 on dist-test-slave-27sr"
server_key: "44edbf91111bf25eb3d10c7cda365151"
server_key_iv: "3d2c4e7ef2a4657d71970c3cf9a84c6e"
server_key_version: "encryptionkey@0"
I20260810 18:06:35.016737 28059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:35.031863 28059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:35.032703 28059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:35.032927 28059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:35.040464 28059 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:34631
I20260810 18:06:35.040470 28134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:34631 every 8 connection(s)
I20260810 18:06:35.041702 28059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:35.042623 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 28059
I20260810 18:06:35.042785 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:35.043124 22233 external_mini_cluster.cc:1428] Setting key 6ec795bb3b31d87499fb2656f01c7b7b
I20260810 18:06:35.045882 28135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:35.052474 28135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565: Bootstrap starting.
I20260810 18:06:35.053722 28139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:35.044718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.21.182.126:34631)
0810 18:06:35.045525 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:35.045542 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:35.045592 (+ 50us) server_negotiation.cc:407] Connection header received
0810 18:06:35.046529 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:35.046551 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:35.046995 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:35.047312 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:35.047707 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.048839 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.050055 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.050565 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.050713 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:35.051760 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:35.051767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:35.051779 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:35.052263 (+ 484us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:35.052323 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:35.052345 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:35.052430 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:35.052642 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 18:06:35.052998 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":148,"threads_started":1}
I20260810 18:06:35.054940 28135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:35.055785 28135 log.cc:824] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:35.056867 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:06:35.057941 28135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565: No bootstrap required, opened a new log
I20260810 18:06:35.061601 28135 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8435fdb5f994cefbab6a2f1c2a36565" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34631 } }
I20260810 18:06:35.061848 28135 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:35.061936 28135 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8435fdb5f994cefbab6a2f1c2a36565, State: Initialized, Role: FOLLOWER
I20260810 18:06:35.062369 28135 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8435fdb5f994cefbab6a2f1c2a36565" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34631 } }
I20260810 18:06:35.062520 28135 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:35.062608 28135 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:35.062728 28135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:35.063637 28135 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8435fdb5f994cefbab6a2f1c2a36565" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34631 } }
I20260810 18:06:35.064059 28135 leader_election.cc:302] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8435fdb5f994cefbab6a2f1c2a36565; no voters:
I20260810 18:06:35.064432 28135 leader_election.cc:288] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:35.064503 28142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:35.064826 28142 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [term 1 LEADER]: Becoming Leader. State: Replica: b8435fdb5f994cefbab6a2f1c2a36565, State: Running, Role: LEADER
I20260810 18:06:35.065150 28142 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8435fdb5f994cefbab6a2f1c2a36565" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34631 } }
I20260810 18:06:35.065742 28135 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:35.066633 28143 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8435fdb5f994cefbab6a2f1c2a36565. Latest consensus state: current_term: 1 leader_uuid: "b8435fdb5f994cefbab6a2f1c2a36565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8435fdb5f994cefbab6a2f1c2a36565" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34631 } } }
I20260810 18:06:35.066905 28144 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8435fdb5f994cefbab6a2f1c2a36565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8435fdb5f994cefbab6a2f1c2a36565" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34631 } } }
I20260810 18:06:35.067013 28143 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:35.067040 28144 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:35.067584 28151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:35.070461 28151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:35.076225 28151 catalog_manager.cc:1347] Generated new cluster ID: a7302105732f47c99d5e3ad1f45fab67
I20260810 18:06:35.076323 28151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:35.111071 28151 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:35.112203 28151 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:35.121912 28151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8435fdb5f994cefbab6a2f1c2a36565: Generated new TSK 0
I20260810 18:06:35.122663 28151 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 18:06:35.151015 28168 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 18:06:35.151212 28168 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 18:06:35.151788 28139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:35.134398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.21.182.126:34631)
0810 18:06:35.134533 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:35.134538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:35.134720 (+ 182us) server_negotiation.cc:407] Connection header received
0810 18:06:35.134953 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:35.134955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:35.135001 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:35.135088 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:06:35.135391 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.135881 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.137190 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.137411 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.137487 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:35.137879 (+ 392us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:06:35.151419 (+ 13540us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 18:06:35.151669 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50318: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 18:06:35.151891 28139 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50318: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 18:06:35.153496 28167 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:35.134250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:34631 (local address 127.0.0.1:50318)
0810 18:06:35.134625 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:35.134641 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:35.134780 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:35.135130 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:35.135134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:35.135143 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:06:35.135280 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:35.135287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.136062 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.136066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:35.137062 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:35.137074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.137528 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.137532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:35.137633 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:35.137689 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:06:35.151684 (+ 13995us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 18:06:35.152001 (+ 317us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.182.126:34631: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.182.126:34631: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 18:06:35.154418 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 28059
2026-08-10T18:06:35Z chronyd exiting
I20260810 18:06:35.195533 22233 test_util.cc:175] -----------------------------------------------
I20260810 18:06:35.195623 22233 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786385164390354-22233-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (355 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 18:06:35.196776 22233 mini_oidc.cc:152] Starting JWKS server
I20260810 18:06:35.197604 22233 webserver.cc:460] Webserver started at https://127.0.0.1:41885/ using document root <none> and password file <none>
I20260810 18:06:35.197682 22233 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 18:06:35.197888 22233 webserver.cc:460] Webserver started at http://127.0.0.1:40523/ using document root <none> and password file <none>
2026-08-10T18:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:35Z Disabled control of system clock
I20260810 18:06:35.208235 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:35949
--webserver_interface=127.21.182.126
--webserver_port=0
--builtin_ntp_servers=127.21.182.84:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:35949
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41885/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/testchainca.cert with env {}
W20260810 18:06:35.322932 28176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:35.323259 28176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:35.323348 28176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:35.327356 28176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:35.327418 28176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:35.327435 28176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:35.327450 28176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:35.327466 28176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:06:35.327481 28176 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:06:35.327502 28176 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41885/jwks.json
I20260810 18:06:35.332782 28176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:35949
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:35949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41885/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:35.333892 28176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:35.334865 28176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:35.343672 28181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:35.343667 28184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:35.343866 28176 server_base.cc:1034] running on GCE node
W20260810 18:06:35.344075 28182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:35.344682 28176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:35.345911 28176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:35.347081 28176 hybrid_clock.cc:648] HybridClock initialized: now 1786385195347063 us; error 29 us; skew 500 ppm
I20260810 18:06:35.349246 28176 webserver.cc:460] Webserver started at http://127.21.182.126:33867/ using document root <none> and password file <none>
I20260810 18:06:35.349884 28176 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:35.349949 28176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:35.350205 28176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:35.352440 28176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "a4d4a62d4eec4129970082efc93f1799"
format_stamp: "Formatted at 2026-08-10 18:06:35 on dist-test-slave-27sr"
server_key: "ad404625d04df64f8790ad74e79da667"
server_key_iv: "f7df20023a61e2a2409ab867dbc203cd"
server_key_version: "encryptionkey@0"
I20260810 18:06:35.353048 28176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "a4d4a62d4eec4129970082efc93f1799"
format_stamp: "Formatted at 2026-08-10 18:06:35 on dist-test-slave-27sr"
server_key: "ad404625d04df64f8790ad74e79da667"
server_key_iv: "f7df20023a61e2a2409ab867dbc203cd"
server_key_version: "encryptionkey@0"
I20260810 18:06:35.357758 28176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:06:35.360630 28190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:35.362207 28176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:35.362393 28176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "a4d4a62d4eec4129970082efc93f1799"
format_stamp: "Formatted at 2026-08-10 18:06:35 on dist-test-slave-27sr"
server_key: "ad404625d04df64f8790ad74e79da667"
server_key_iv: "f7df20023a61e2a2409ab867dbc203cd"
server_key_version: "encryptionkey@0"
I20260810 18:06:35.362550 28176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:35.375548 28176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:35.376408 28176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:35.376621 28176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:35.384569 28176 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:35949
I20260810 18:06:35.384578 28255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:35949 every 8 connection(s)
I20260810 18:06:35.385651 28176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:35.389149 28256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:35.394730 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 28176
I20260810 18:06:35.394884 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:35.395200 22233 external_mini_cluster.cc:1428] Setting key 876a6c0ffa67dc65adba875ecdb78c4d
I20260810 18:06:35.395500 28256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799: Bootstrap starting.
I20260810 18:06:35.398509 28256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:35.399312 28256 log.cc:824] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:35.401428 28256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799: No bootstrap required, opened a new log
I20260810 18:06:35.405273 28256 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d4a62d4eec4129970082efc93f1799" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35949 } }
I20260810 18:06:35.405560 28256 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:35.405630 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:35.396804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.21.182.126:35949)
0810 18:06:35.397636 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:35.397722 (+ 86us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:35.397765 (+ 43us) server_negotiation.cc:407] Connection header received
0810 18:06:35.398497 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:35.398542 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:35.399055 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:35.399416 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:35.399939 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.400941 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.401955 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.402459 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.402646 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:35.403595 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:35.403600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:35.403615 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:35.404148 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:35.404212 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:35.404236 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:35.404351 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:35.404476 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 18:06:35.404775 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":163,"threads_started":1}
I20260810 18:06:35.405646 28256 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d4a62d4eec4129970082efc93f1799, State: Initialized, Role: FOLLOWER
I20260810 18:06:35.406230 28256 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d4a62d4eec4129970082efc93f1799" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35949 } }
I20260810 18:06:35.406414 28256 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:35.406481 28256 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:35.406580 28256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:35.407682 28256 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d4a62d4eec4129970082efc93f1799" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35949 } }
I20260810 18:06:35.408095 28256 leader_election.cc:302] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4d4a62d4eec4129970082efc93f1799; no voters:
I20260810 18:06:35.408564 28256 leader_election.cc:288] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:35.408684 28262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:35.408916 28262 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [term 1 LEADER]: Becoming Leader. State: Replica: a4d4a62d4eec4129970082efc93f1799, State: Running, Role: LEADER
I20260810 18:06:35.409139 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:06:35.409261 28262 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d4a62d4eec4129970082efc93f1799" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35949 } }
I20260810 18:06:35.410043 28256 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:35.410672 28264 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4d4a62d4eec4129970082efc93f1799. Latest consensus state: current_term: 1 leader_uuid: "a4d4a62d4eec4129970082efc93f1799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d4a62d4eec4129970082efc93f1799" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35949 } } }
I20260810 18:06:35.410799 28264 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:35.411154 28263 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4d4a62d4eec4129970082efc93f1799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d4a62d4eec4129970082efc93f1799" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35949 } } }
I20260810 18:06:35.411273 28263 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799 [sys.catalog]: This master's current role is: LEADER
W20260810 18:06:35.415557 28278 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:06:35.415676 28278 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:06:35.415756 28267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:35.416574 28267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:35.423098 28267 catalog_manager.cc:1347] Generated new cluster ID: 604a1272ed3d41f19b686f12e6fb6ea5
I20260810 18:06:35.423198 28267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:35.454209 28267 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:35.455673 28267 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:35.469543 28267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4d4a62d4eec4129970082efc93f1799: Generated new TSK 0
I20260810 18:06:35.470333 28267 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 18:06:35.505899 28290 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 18:06:35.506007 28290 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 18:06:35.506824 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:35.490048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57882 (local address 127.21.182.126:35949)
0810 18:06:35.490204 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:35.490211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:35.490388 (+ 177us) server_negotiation.cc:407] Connection header received
0810 18:06:35.490610 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:35.490613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:35.490668 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:35.490762 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:06:35.491078 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.491612 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.492917 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.493142 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.493226 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:35.493731 (+ 505us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:06:35.506313 (+ 12582us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41885/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 18:06:35.506660 (+ 347us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57882: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41885/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 18:06:35.506901 28260 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57882: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41885/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 18:06:35.506938 28289 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:35.489950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35949 (local address 127.0.0.1:57882)
0810 18:06:35.490282 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:35.490299 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:35.490444 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:35.490795 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:35.490799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:35.490807 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:06:35.490942 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:35.490949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.491746 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.491750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:35.492784 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:35.492797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.493268 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.493272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:35.493400 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:35.493432 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:06:35.506562 (+ 13130us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41885/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 18:06:35.506748 (+ 186us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.182.126:35949: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41885/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":115,"threads_started":1}
I20260810 18:06:35.507571 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 28176
2026-08-10T18:06:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (355 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 18:06:35.552672 22233 mini_oidc.cc:152] Starting JWKS server
I20260810 18:06:35.553687 22233 webserver.cc:460] Webserver started at https://127.0.0.1:32877/ using document root <none> and password file <none>
I20260810 18:06:35.553845 22233 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 18:06:35.554158 22233 webserver.cc:460] Webserver started at http://127.0.0.1:46485/ using document root <none> and password file <none>
2026-08-10T18:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:35Z Disabled control of system clock
I20260810 18:06:35.565927 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:39703
--webserver_interface=127.21.182.126
--webserver_port=0
--builtin_ntp_servers=127.21.182.84:44941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:39703
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32877/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 18:06:35.686954 28296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:35.687271 28296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:35.687312 28296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:35.691500 28296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:35.691586 28296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:35.691601 28296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:35.691619 28296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:35.691635 28296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:06:35.691694 28296 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:06:35.691715 28296 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32877/jwks.json
I20260810 18:06:35.696931 28296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:44941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:39703
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:39703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32877/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:35.698194 28296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:35.699712 28296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:35.709162 28301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:35.709151 28302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:35.709151 28304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:35.709367 28296 server_base.cc:1034] running on GCE node
I20260810 18:06:35.710130 28296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:35.711505 28296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:35.712718 28296 hybrid_clock.cc:648] HybridClock initialized: now 1786385195712691 us; error 36 us; skew 500 ppm
I20260810 18:06:35.714952 28296 webserver.cc:460] Webserver started at http://127.21.182.126:39125/ using document root <none> and password file <none>
I20260810 18:06:35.715602 28296 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:35.715674 28296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:35.715965 28296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:35.718286 28296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "a062cd159b044a978e483092ed65b838"
format_stamp: "Formatted at 2026-08-10 18:06:35 on dist-test-slave-27sr"
server_key: "37dca680a5143a0628943966fe3dfeef"
server_key_iv: "5ae2aeadcb4025ef0bf6c1379cb84e2c"
server_key_version: "encryptionkey@0"
I20260810 18:06:35.719003 28296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "a062cd159b044a978e483092ed65b838"
format_stamp: "Formatted at 2026-08-10 18:06:35 on dist-test-slave-27sr"
server_key: "37dca680a5143a0628943966fe3dfeef"
server_key_iv: "5ae2aeadcb4025ef0bf6c1379cb84e2c"
server_key_version: "encryptionkey@0"
I20260810 18:06:35.723218 28296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:06:35.725888 28310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:35.727223 28296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:06:35.727396 28296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "a062cd159b044a978e483092ed65b838"
format_stamp: "Formatted at 2026-08-10 18:06:35 on dist-test-slave-27sr"
server_key: "37dca680a5143a0628943966fe3dfeef"
server_key_iv: "5ae2aeadcb4025ef0bf6c1379cb84e2c"
server_key_version: "encryptionkey@0"
I20260810 18:06:35.727526 28296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:35.746395 28296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:35.747232 28296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:35.747471 28296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:35.755309 28296 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:39703
I20260810 18:06:35.755371 28371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:39703 every 8 connection(s)
I20260810 18:06:35.756465 28296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:35.759773 28372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:35.763510 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 28296
I20260810 18:06:35.763675 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:35.764056 22233 external_mini_cluster.cc:1428] Setting key 1df68caa8f3e102c02be134cd417d4c5
I20260810 18:06:35.766420 28372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838: Bootstrap starting.
I20260810 18:06:35.769100 28372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:35.770084 28372 log.cc:824] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:35.772706 28372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838: No bootstrap required, opened a new log
I20260810 18:06:35.774484 28375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:35.765660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.21.182.126:39703)
0810 18:06:35.766367 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:35.766384 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:35.766422 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:06:35.767143 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:35.767160 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:35.767607 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:35.767970 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:35.768354 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.769413 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.770654 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.771329 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.771455 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:35.772415 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:35.772420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:35.772434 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:35.772900 (+ 466us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:35.772960 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:35.772981 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:35.773062 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:35.773292 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 18:06:35.773627 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":158,"threads_started":1}
I20260810 18:06:35.776878 28372 raft_consensus.cc:348] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a062cd159b044a978e483092ed65b838" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39703 } }
I20260810 18:06:35.777112 28372 raft_consensus.cc:374] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:35.777189 28372 raft_consensus.cc:729] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a062cd159b044a978e483092ed65b838, State: Initialized, Role: FOLLOWER
I20260810 18:06:35.777675 28372 consensus_queue.cc:260] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a062cd159b044a978e483092ed65b838" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39703 } }
I20260810 18:06:35.777848 28372 raft_consensus.cc:388] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:35.777938 28372 raft_consensus.cc:482] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:35.778013 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:06:35.778041 28372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:35.779040 28372 raft_consensus.cc:504] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a062cd159b044a978e483092ed65b838" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39703 } }
I20260810 18:06:35.779431 28372 leader_election.cc:302] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a062cd159b044a978e483092ed65b838; no voters:
I20260810 18:06:35.779830 28372 leader_election.cc:288] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:35.779932 28377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:35.780149 28377 raft_consensus.cc:686] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [term 1 LEADER]: Becoming Leader. State: Replica: a062cd159b044a978e483092ed65b838, State: Running, Role: LEADER
I20260810 18:06:35.780472 28377 consensus_queue.cc:237] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a062cd159b044a978e483092ed65b838" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39703 } }
I20260810 18:06:35.781263 28372 sys_catalog.cc:564] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:35.782377 28378 sys_catalog.cc:455] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a062cd159b044a978e483092ed65b838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a062cd159b044a978e483092ed65b838" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39703 } } }
I20260810 18:06:35.782434 28379 sys_catalog.cc:455] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a062cd159b044a978e483092ed65b838. Latest consensus state: current_term: 1 leader_uuid: "a062cd159b044a978e483092ed65b838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a062cd159b044a978e483092ed65b838" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39703 } } }
I20260810 18:06:35.782613 28379 sys_catalog.cc:458] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:35.782648 28378 sys_catalog.cc:458] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:35.783067 28388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:35.786966 28388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:35.794044 28388 catalog_manager.cc:1347] Generated new cluster ID: 76afcf9752ba4c3b80fa3577e8833863
I20260810 18:06:35.794147 28388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:35.805701 28388 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:35.807164 28388 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:35.816412 28388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a062cd159b044a978e483092ed65b838: Generated new TSK 0
I20260810 18:06:35.817461 28388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:06:35.835013 28375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:35.829256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.21.182.126:39703)
0810 18:06:35.829407 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:35.829412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:35.829602 (+ 190us) server_negotiation.cc:407] Connection header received
0810 18:06:35.829817 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:35.829820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:35.829896 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:35.830002 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:06:35.830327 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.830993 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.832453 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.832744 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.832835 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:35.833227 (+ 392us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:06:35.834586 (+ 1359us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 18:06:35.834874 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43486: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 18:06:35.835109 28375 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43486: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 18:06:35.837890 28405 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:35.829094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:39703 (local address 127.0.0.1:43486)
0810 18:06:35.829490 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:35.829506 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:35.829655 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:35.830047 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:35.830050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:35.830060 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:06:35.830201 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:35.830209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.831144 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.831149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:35.832318 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:35.832332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:35.832895 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:35.832899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:35.832996 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:35.833030 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:06:35.834762 (+ 1732us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 18:06:35.837728 (+ 2966us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.182.126:39703: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32877/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":2746,"thread_start_us":134,"threads_started":1}
I20260810 18:06:35.838601 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 28296
E20260810 18:06:35.849226 28406 webserver.cc:533] Webserver: sslize failed:
E20260810 18:06:35.849318 28406 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T18:06:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (330 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T18:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:35Z Disabled control of system clock
I20260810 18:06:35.893916 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:34665
--webserver_interface=127.21.182.126
--webserver_port=0
--builtin_ntp_servers=127.21.182.84:36139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:34665
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/testchainca.cert with env {}
W20260810 18:06:36.010676 28410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:36.011039 28410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:36.011130 28410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:36.015306 28410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:36.015426 28410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:36.015453 28410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:36.015483 28410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:36.015523 28410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:06:36.015560 28410 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:06:36.015588 28410 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 18:06:36.021427 28410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:36139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:34665
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:34665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:36.022655 28410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:36.023762 28410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:36.032353 28416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:36.032415 28415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:36.032361 28418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:36.032586 28410 server_base.cc:1034] running on GCE node
I20260810 18:06:36.033232 28410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:36.034411 28410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:36.035605 28410 hybrid_clock.cc:648] HybridClock initialized: now 1786385196035578 us; error 40 us; skew 500 ppm
I20260810 18:06:36.037549 28410 webserver.cc:460] Webserver started at http://127.21.182.126:37581/ using document root <none> and password file <none>
I20260810 18:06:36.038152 28410 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:36.038249 28410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:36.038496 28410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:36.040724 28410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "9d2e61bc96a74ea998c200e636a7813f"
format_stamp: "Formatted at 2026-08-10 18:06:36 on dist-test-slave-27sr"
server_key: "9ba4f5efe44a7a027d927c3d05e5a31d"
server_key_iv: "130ec31db69812f185df3140fa63a968"
server_key_version: "encryptionkey@0"
I20260810 18:06:36.041345 28410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "9d2e61bc96a74ea998c200e636a7813f"
format_stamp: "Formatted at 2026-08-10 18:06:36 on dist-test-slave-27sr"
server_key: "9ba4f5efe44a7a027d927c3d05e5a31d"
server_key_iv: "130ec31db69812f185df3140fa63a968"
server_key_version: "encryptionkey@0"
I20260810 18:06:36.045344 28410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:06:36.048106 28424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:36.049386 28410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:36.049538 28410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "9d2e61bc96a74ea998c200e636a7813f"
format_stamp: "Formatted at 2026-08-10 18:06:36 on dist-test-slave-27sr"
server_key: "9ba4f5efe44a7a027d927c3d05e5a31d"
server_key_iv: "130ec31db69812f185df3140fa63a968"
server_key_version: "encryptionkey@0"
I20260810 18:06:36.049671 28410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:36.069355 28410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:36.070191 28410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:36.070431 28410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:36.077867 28410 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:34665
I20260810 18:06:36.077876 28479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:34665 every 8 connection(s)
I20260810 18:06:36.078919 28410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:36.080828 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 28410
I20260810 18:06:36.080971 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:36.081292 22233 external_mini_cluster.cc:1428] Setting key b18edfc5ce60502857b856172fcf8937
I20260810 18:06:36.082823 28480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:36.088734 28480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f: Bootstrap starting.
I20260810 18:06:36.090770 28483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:36.082788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.21.182.126:34665)
0810 18:06:36.083482 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:36.083503 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:36.083540 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:06:36.084266 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:36.084286 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:36.084695 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:36.085055 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:36.085496 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:36.086470 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:36.087381 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:36.087854 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:36.088107 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:36.089069 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:36.089074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:36.089086 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:06:36.089477 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:36.089534 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:06:36.089555 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:36.089670 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:36.089791 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 18:06:36.090058 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":151,"threads_started":1}
I20260810 18:06:36.091209 28480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:36.092130 28480 log.cc:824] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:36.093994 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:06:36.094451 28480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f: No bootstrap required, opened a new log
I20260810 18:06:36.098271 28480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2e61bc96a74ea998c200e636a7813f" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34665 } }
I20260810 18:06:36.098558 28480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:36.098625 28480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2e61bc96a74ea998c200e636a7813f, State: Initialized, Role: FOLLOWER
I20260810 18:06:36.099136 28480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2e61bc96a74ea998c200e636a7813f" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34665 } }
I20260810 18:06:36.099292 28480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:36.099385 28480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:36.099503 28480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:36.100545 28480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2e61bc96a74ea998c200e636a7813f" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34665 } }
I20260810 18:06:36.100898 28480 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d2e61bc96a74ea998c200e636a7813f; no voters:
I20260810 18:06:36.101233 28480 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:36.101370 28485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:36.101603 28485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [term 1 LEADER]: Becoming Leader. State: Replica: 9d2e61bc96a74ea998c200e636a7813f, State: Running, Role: LEADER
I20260810 18:06:36.101931 28485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2e61bc96a74ea998c200e636a7813f" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34665 } }
I20260810 18:06:36.102633 28480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:36.103809 28488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2e61bc96a74ea998c200e636a7813f. Latest consensus state: current_term: 1 leader_uuid: "9d2e61bc96a74ea998c200e636a7813f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2e61bc96a74ea998c200e636a7813f" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34665 } } }
I20260810 18:06:36.104072 28486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d2e61bc96a74ea998c200e636a7813f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2e61bc96a74ea998c200e636a7813f" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34665 } } }
I20260810 18:06:36.104221 28486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:36.104180 28488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:36.105183 28497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:36.108057 28497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:36.115083 28497 catalog_manager.cc:1347] Generated new cluster ID: f988316514494118abe6f4fcbcebf0ea
I20260810 18:06:36.115209 28497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:36.131609 28497 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:36.132659 28497 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:36.138796 28497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d2e61bc96a74ea998c200e636a7813f: Generated new TSK 0
I20260810 18:06:36.139524 28497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:06:36.594691 28483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:36.152723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.21.182.126:34665)
0810 18:06:36.152879 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:36.152884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:36.153011 (+ 127us) server_negotiation.cc:407] Connection header received
0810 18:06:36.153196 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:36.153198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:36.153248 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:36.153380 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:06:36.153669 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:36.154336 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:36.155656 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:36.155942 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:36.156034 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:36.156431 (+ 397us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:06:36.594218 (+437787us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 18:06:36.594554 (+ 336us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54532: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 18:06:36.594779 28483 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54532: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 18:06:36.595039 28513 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:36.152556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:34665 (local address 127.0.0.1:54532)
0810 18:06:36.152910 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:36.152926 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:36.153050 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:36.153423 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:36.153427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:36.153440 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:06:36.153559 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:36.153565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:36.154517 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:36.154521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:36.155523 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:36.155534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:36.156074 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:36.156078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:36.156176 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:36.156214 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:06:36.594524 (+438310us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 18:06:36.594802 (+ 278us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.182.126:34665: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":127,"threads_started":1}
I20260810 18:06:36.595791 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 28410
2026-08-10T18:06:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (738 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (139 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (141 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:06:36 dist-test-slave-27sr krb5kdc[28525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:06:36 dist-test-slave-27sr krb5kdc[28525](info): set up 2 sockets
Aug 10 18:06:36 dist-test-slave-27sr krb5kdc[28525](info): commencing operation
krb5kdc: starting...
W20260810 18:06:38.074050 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:06:38 dist-test-slave-27sr krb5kdc[28525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:06:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:06:38.215873 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:39241
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:39241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:38.332947 28539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:38.333283 28539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:38.333369 28539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:38.337476 28539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:38.337581 28539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:38.337622 28539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:38.337661 28539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:38.337687 28539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:38.343135 28539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:39241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:39241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:38.344357 28539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:38.345459 28539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:38.354750 28546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:38.354823 28539 server_base.cc:1034] running on GCE node
W20260810 18:06:38.354750 28547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:38.354779 28550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:38.355604 28539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:38.356986 28539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:38.358206 28539 hybrid_clock.cc:648] HybridClock initialized: now 1786385198358190 us; error 34 us; skew 500 ppm
Aug 10 18:06:38 dist-test-slave-27sr krb5kdc[28525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385198, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:38.364593 28539 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:06:38.365669 28539 webserver.cc:460] Webserver started at http://127.21.182.126:44787/ using document root <none> and password file <none>
I20260810 18:06:38.366237 28539 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:38.366299 28539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:38.366479 28539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:38.368703 28539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "f9981d5e292149e9bd2e14d710a28dfb"
format_stamp: "Formatted at 2026-08-10 18:06:38 on dist-test-slave-27sr"
server_key: "f78e4258de65903461813b2d64e68f52"
server_key_iv: "09f110e3bbe2ab23acf3adca42e369d4"
server_key_version: "encryptionkey@0"
I20260810 18:06:38.369267 28539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "f9981d5e292149e9bd2e14d710a28dfb"
format_stamp: "Formatted at 2026-08-10 18:06:38 on dist-test-slave-27sr"
server_key: "f78e4258de65903461813b2d64e68f52"
server_key_iv: "09f110e3bbe2ab23acf3adca42e369d4"
server_key_version: "encryptionkey@0"
I20260810 18:06:38.373304 28539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:06:38.376001 28558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:38.377363 28539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:38.377507 28539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "f9981d5e292149e9bd2e14d710a28dfb"
format_stamp: "Formatted at 2026-08-10 18:06:38 on dist-test-slave-27sr"
server_key: "f78e4258de65903461813b2d64e68f52"
server_key_iv: "09f110e3bbe2ab23acf3adca42e369d4"
server_key_version: "encryptionkey@0"
I20260810 18:06:38.377635 28539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:38.411644 28539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:38.415140 28539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:38.415387 28539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:38.423503 28539 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:39241
I20260810 18:06:38.423523 28619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:39241 every 8 connection(s)
I20260810 18:06:38.424686 28539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:38.428025 28620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:38.434334 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 28539
I20260810 18:06:38.434496 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:38.434782 28620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb: Bootstrap starting.
I20260810 18:06:38.434840 22233 external_mini_cluster.cc:1428] Setting key dda46872f44fba1e4bab11074ecca578
I20260810 18:06:38.437857 28620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:38.438882 28620 log.cc:824] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:38.441138 28620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb: No bootstrap required, opened a new log
Aug 10 18:06:38 dist-test-slave-27sr krb5kdc[28525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:38.445415 28620 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9981d5e292149e9bd2e14d710a28dfb" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39241 } }
I20260810 18:06:38.445683 28620 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:38.445720 28620 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9981d5e292149e9bd2e14d710a28dfb, State: Initialized, Role: FOLLOWER
I20260810 18:06:38.446334 28620 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9981d5e292149e9bd2e14d710a28dfb" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39241 } }
I20260810 18:06:38.446516 28620 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:38.446611 28620 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:38.446740 28620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:38.447809 28620 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9981d5e292149e9bd2e14d710a28dfb" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39241 } }
I20260810 18:06:38.448254 28620 leader_election.cc:302] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9981d5e292149e9bd2e14d710a28dfb; no voters:
I20260810 18:06:38.448716 28620 leader_election.cc:288] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:38.448995 28625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:38.449286 28625 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [term 1 LEADER]: Becoming Leader. State: Replica: f9981d5e292149e9bd2e14d710a28dfb, State: Running, Role: LEADER
I20260810 18:06:38.449692 28625 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9981d5e292149e9bd2e14d710a28dfb" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39241 } }
I20260810 18:06:38.450345 28620 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:38.451177 28625 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9981d5e292149e9bd2e14d710a28dfb. Latest consensus state: current_term: 1 leader_uuid: "f9981d5e292149e9bd2e14d710a28dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9981d5e292149e9bd2e14d710a28dfb" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39241 } } }
I20260810 18:06:38.451404 28625 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:38.451728 28626 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9981d5e292149e9bd2e14d710a28dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9981d5e292149e9bd2e14d710a28dfb" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39241 } } }
I20260810 18:06:38.451956 28626 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:38.451952 28634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:38.452929 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:38.436417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52282 (local address 127.21.182.126:39241)
0810 18:06:38.436920 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:38.436938 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:38.436979 (+ 41us) server_negotiation.cc:407] Connection header received
0810 18:06:38.437682 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:38.437701 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:38.438145 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:38.438465 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:38.439549 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:38.440621 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:38.441588 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:38.442088 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:38.442301 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:38.445411 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:38.445417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:38.445445 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:38.445481 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:38.447572 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:38.448233 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:38.448240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:38.448247 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:38.448411 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:38.448871 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:38.448877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:38.448896 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:38.449585 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:38.449817 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:38.450375 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 18:06:38.450739 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":166,"threads_started":1}
I20260810 18:06:38.455965 28634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:38.462827 28634 catalog_manager.cc:1347] Generated new cluster ID: 25a758ec49034764a211d9e43b2f9951
I20260810 18:06:38.462950 28634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:38.475277 28634 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:38.476485 28634 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:38.482568 28634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9981d5e292149e9bd2e14d710a28dfb: Generated new TSK 0
I20260810 18:06:38.483440 28634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:06:38.520548 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39241
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:38.656884 28648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:38.657207 28648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:38.657258 28648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:38.661908 28648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:38.661993 28648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:38.662057 28648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:38.667796 28648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:38.669028 28648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:38.670112 28648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:06:38.680866 28648 server_base.cc:1034] running on GCE node
W20260810 18:06:38.680734 28657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:38.680811 28654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:38.680734 28653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:38.681566 28648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:38.682394 28648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:38.683588 28648 hybrid_clock.cc:648] HybridClock initialized: now 1786385198683559 us; error 47 us; skew 500 ppm
Aug 10 18:06:38 dist-test-slave-27sr krb5kdc[28525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385198, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:38.686678 28648 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:06:38.687601 28648 webserver.cc:460] Webserver started at http://127.21.182.65:37823/ using document root <none> and password file <none>
I20260810 18:06:38.688253 28648 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:38.688323 28648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:38.688514 28648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:38.690791 28648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "cd863e19a57541559ecee24f14514d22"
format_stamp: "Formatted at 2026-08-10 18:06:38 on dist-test-slave-27sr"
server_key: "2719a3e633947f14c587213b2b1b9635"
server_key_iv: "aebae4691bd84adc37635db357490d53"
server_key_version: "encryptionkey@0"
I20260810 18:06:38.691418 28648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "cd863e19a57541559ecee24f14514d22"
format_stamp: "Formatted at 2026-08-10 18:06:38 on dist-test-slave-27sr"
server_key: "2719a3e633947f14c587213b2b1b9635"
server_key_iv: "aebae4691bd84adc37635db357490d53"
server_key_version: "encryptionkey@0"
I20260810 18:06:38.695511 28648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:06:38.698199 28665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:38.699491 28648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:06:38.699626 28648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "cd863e19a57541559ecee24f14514d22"
format_stamp: "Formatted at 2026-08-10 18:06:38 on dist-test-slave-27sr"
server_key: "2719a3e633947f14c587213b2b1b9635"
server_key_iv: "aebae4691bd84adc37635db357490d53"
server_key_version: "encryptionkey@0"
I20260810 18:06:38.699745 28648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:38.710441 28648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:38.713837 28648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:38.714061 28648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:38.714748 28648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:38.715723 28648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:38.715775 28648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:38.715818 28648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:38.715833 28648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:38.725981 28648 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:34241
I20260810 18:06:38.726019 28790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:34241 every 8 connection(s)
I20260810 18:06:38.726984 28648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:38.729548 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 28648
I20260810 18:06:38.729694 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:06:38.730059 22233 external_mini_cluster.cc:1428] Setting key 0d3389cc19be553eefad0b110131bc1f
Aug 10 18:06:38 dist-test-slave-27sr krb5kdc[28525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385198, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
I20260810 18:06:38.744026 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:38.729147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:37315 (local address 127.21.182.126:39241)
0810 18:06:38.729328 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:38.729331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:38.730493 (+ 1162us) server_negotiation.cc:407] Connection header received
0810 18:06:38.732377 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:38.732384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:38.732451 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:38.732551 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:38.733796 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:38.734466 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:38.736296 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:38.736540 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:38.736636 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:38.739874 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:38.739881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:38.739886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:38.739956 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:38.741674 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:38.742273 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:38.742280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:38.742285 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:38.742395 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:38.742785 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:38.742788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:38.742789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:38.742989 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:38.743074 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:38.743700 (+ 626us) server_negotiation.cc:299] Negotiation successful
0810 18:06:38.743826 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 18:06:38.744973 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:38.729697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:39241 (local address 127.21.182.65:37315)
0810 18:06:38.730317 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:38.730367 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:38.732136 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:38.732722 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:38.732729 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:38.733129 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:38.733630 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:38.733643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:38.734603 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:38.734608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:38.736120 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:38.736135 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:38.736714 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:38.736721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:38.737006 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:38.737795 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:38.737822 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:38.739671 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:38.741811 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:38.741817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:38.741832 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:38.742152 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:38.742523 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:38.742527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:38.742528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:38.742693 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:38.743116 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:38.743122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:38.743457 (+ 335us) client_negotiation.cc:769] Sending connection context
0810 18:06:38.743639 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 18:06:38.744046 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":159,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
I20260810 18:06:38.746409 28791 heartbeater.cc:347] Connected to a master server at 127.21.182.126:39241
I20260810 18:06:38.746697 28791 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:38.747380 28791 heartbeater.cc:511] Master 127.21.182.126:39241 requested a full tablet report, sending...
I20260810 18:06:38.749279 28578 ts_manager.cc:194] Registered new tserver with Master: cd863e19a57541559ecee24f14514d22 (127.21.182.65:34241)
I20260810 18:06:38.750679 28578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:37315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:06:38.763444 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39241
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:38.885659 28796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:38.885933 28796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:38.885980 28796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:38.890159 28796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:38.890219 28796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:38.890276 28796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:38.895516 28796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:38.896610 28796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:38.897610 28796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:38.907891 28802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:38.907977 28801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:38.907931 28804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:38.908771 28796 server_base.cc:1034] running on GCE node
I20260810 18:06:38.909250 28796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:38.909966 28796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:38.911163 28796 hybrid_clock.cc:648] HybridClock initialized: now 1786385198911133 us; error 47 us; skew 500 ppm
Aug 10 18:06:38 dist-test-slave-27sr krb5kdc[28525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385198, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:38.914259 28796 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:06:38.915400 28796 webserver.cc:460] Webserver started at http://127.21.182.66:44553/ using document root <none> and password file <none>
I20260810 18:06:38.916162 28796 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:38.916267 28796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:38.916538 28796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:38.919035 28796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "6b1d7782c0eb49d282bf6896b56c8ab5"
format_stamp: "Formatted at 2026-08-10 18:06:38 on dist-test-slave-27sr"
server_key: "f5274949e344a88ea5eba6ea25b7b4b1"
server_key_iv: "a3d68a183a9ef615cdc3c6d3f8d2df42"
server_key_version: "encryptionkey@0"
I20260810 18:06:38.919728 28796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "6b1d7782c0eb49d282bf6896b56c8ab5"
format_stamp: "Formatted at 2026-08-10 18:06:38 on dist-test-slave-27sr"
server_key: "f5274949e344a88ea5eba6ea25b7b4b1"
server_key_iv: "a3d68a183a9ef615cdc3c6d3f8d2df42"
server_key_version: "encryptionkey@0"
I20260810 18:06:38.924155 28796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:06:38.926872 28811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:38.928210 28796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:38.928359 28796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "6b1d7782c0eb49d282bf6896b56c8ab5"
format_stamp: "Formatted at 2026-08-10 18:06:38 on dist-test-slave-27sr"
server_key: "f5274949e344a88ea5eba6ea25b7b4b1"
server_key_iv: "a3d68a183a9ef615cdc3c6d3f8d2df42"
server_key_version: "encryptionkey@0"
I20260810 18:06:38.928489 28796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:38.943768 28796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:38.946787 28796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:38.947021 28796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:38.947649 28796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:38.948593 28796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:38.948665 28796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:38.948732 28796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:38.948762 28796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:38.958436 28796 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:42381
I20260810 18:06:38.958454 28939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:42381 every 8 connection(s)
I20260810 18:06:38.959393 28796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:06:38.962541 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 28796
I20260810 18:06:38.962683 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:06:38.963007 22233 external_mini_cluster.cc:1428] Setting key df0d6363c96e82a48fc18cc00f9d9e9b
Aug 10 18:06:38 dist-test-slave-27sr krb5kdc[28525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385198, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
I20260810 18:06:38.976709 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:38.961417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:37985 (local address 127.21.182.126:39241)
0810 18:06:38.961561 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:38.961565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:38.962779 (+ 1214us) server_negotiation.cc:407] Connection header received
0810 18:06:38.964737 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:38.964741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:38.964798 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:38.964852 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:38.966830 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:38.967396 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:38.969224 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:38.969445 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:38.969533 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:38.972769 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:38.972773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:38.972775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:38.972803 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:38.974336 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:38.974923 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:38.974926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:38.974928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:38.975012 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:38.975416 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:38.975419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:38.975421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:38.975633 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:38.975750 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:38.976456 (+ 706us) server_negotiation.cc:299] Negotiation successful
0810 18:06:38.976576 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:06:38.977766 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:38.961900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:39241 (local address 127.21.182.66:37985)
0810 18:06:38.962501 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:38.962565 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:38.964465 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:38.965285 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:38.965296 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:38.965890 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:38.966640 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:38.966663 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:38.967542 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:38.967548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:38.969083 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:38.969098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:38.969615 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:38.969621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:38.969875 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:38.970664 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:38.970693 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:38.972556 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:38.974481 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:38.974488 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:38.974507 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:38.974790 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:38.975111 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:38.975114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:38.975116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:38.975298 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:38.975749 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:38.975757 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:38.976121 (+ 364us) client_negotiation.cc:769] Sending connection context
0810 18:06:38.976379 (+ 258us) client_negotiation.cc:240] Negotiation successful
0810 18:06:38.976696 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":151,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
I20260810 18:06:38.979303 28940 heartbeater.cc:347] Connected to a master server at 127.21.182.126:39241
I20260810 18:06:38.979600 28940 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:38.980193 28940 heartbeater.cc:511] Master 127.21.182.126:39241 requested a full tablet report, sending...
I20260810 18:06:38.981551 28578 ts_manager.cc:194] Registered new tserver with Master: 6b1d7782c0eb49d282bf6896b56c8ab5 (127.21.182.66:42381)
I20260810 18:06:38.982357 28578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:37985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:06:38.996711 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39241
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:39.116312 28945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:39.116644 28945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:39.116729 28945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:39.120980 28945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:39.121083 28945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:39.121174 28945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:39.126642 28945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:39.127844 28945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:39.128996 28945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:39.139266 28951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:39.139299 28952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:39.139459 28945 server_base.cc:1034] running on GCE node
W20260810 18:06:39.139292 28954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:39.140025 28945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:39.140717 28945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:39.141912 28945 hybrid_clock.cc:648] HybridClock initialized: now 1786385199141897 us; error 28 us; skew 500 ppm
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[28525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385199, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:39.144763 28945 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:06:39.145718 28945 webserver.cc:460] Webserver started at http://127.21.182.67:36759/ using document root <none> and password file <none>
I20260810 18:06:39.146309 28945 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:39.146416 28945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:39.146660 28945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:39.148926 28945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "66587d6e1a0248afa176aa20b8989813"
format_stamp: "Formatted at 2026-08-10 18:06:39 on dist-test-slave-27sr"
server_key: "3e1d3a3b3a4ca05118ac2fcf4257e3c8"
server_key_iv: "987074176b2f6acb570c0f856cbc87dd"
server_key_version: "encryptionkey@0"
I20260810 18:06:39.149494 28945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "66587d6e1a0248afa176aa20b8989813"
format_stamp: "Formatted at 2026-08-10 18:06:39 on dist-test-slave-27sr"
server_key: "3e1d3a3b3a4ca05118ac2fcf4257e3c8"
server_key_iv: "987074176b2f6acb570c0f856cbc87dd"
server_key_version: "encryptionkey@0"
I20260810 18:06:39.153643 28945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 18:06:39.156239 28962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:39.157524 28945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:06:39.157663 28945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "66587d6e1a0248afa176aa20b8989813"
format_stamp: "Formatted at 2026-08-10 18:06:39 on dist-test-slave-27sr"
server_key: "3e1d3a3b3a4ca05118ac2fcf4257e3c8"
server_key_iv: "987074176b2f6acb570c0f856cbc87dd"
server_key_version: "encryptionkey@0"
I20260810 18:06:39.157778 28945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:39.173725 28945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:39.176903 28945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:39.177120 28945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:39.177753 28945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:39.178632 28945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:39.178720 28945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:39.178794 28945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:39.178843 28945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:39.187731 28945 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:42453
I20260810 18:06:39.187800 29085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:42453 every 8 connection(s)
I20260810 18:06:39.188696 28945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:06:39.194996 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 28945
I20260810 18:06:39.195139 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:06:39.195513 22233 external_mini_cluster.cc:1428] Setting key 1437101110668a7b328605e5687dc9e2
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[28525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385199, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:39.205156 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:39.190591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:57821 (local address 127.21.182.126:39241)
0810 18:06:39.190759 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:39.190764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:39.191701 (+ 937us) server_negotiation.cc:407] Connection header received
0810 18:06:39.193091 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:39.193094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:39.193143 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:39.193222 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:39.194444 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:39.195289 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:39.196685 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:39.197109 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:39.197238 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:39.201154 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:39.201157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:39.201160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:39.201188 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:39.202923 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:39.203430 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:39.203433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:39.203435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:39.203520 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:39.203854 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:39.203857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:39.203858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:39.204137 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:39.204252 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:39.204907 (+ 655us) server_negotiation.cc:299] Negotiation successful
0810 18:06:39.205021 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 18:06:39.205952 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:39.191048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:39241 (local address 127.21.182.67:57821)
0810 18:06:39.191532 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:39.191575 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:39.192888 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:39.193394 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:39.193401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:39.193756 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:39.194272 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:39.194284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:39.195430 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:39.195433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:39.196511 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:39.196522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:39.198170 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:39.198177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:39.198403 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:39.199195 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:39.199212 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:39.200986 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:39.203030 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:39.203035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:39.203049 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:39.203301 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:39.203604 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:39.203607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:39.203608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:39.203746 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:39.204262 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:39.204272 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:39.204574 (+ 302us) client_negotiation.cc:769] Sending connection context
0810 18:06:39.204738 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 18:06:39.205020 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":129,"threads_started":1}
I20260810 18:06:39.207275 29086 heartbeater.cc:347] Connected to a master server at 127.21.182.126:39241
I20260810 18:06:39.207521 29086 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:39.208088 29086 heartbeater.cc:511] Master 127.21.182.126:39241 requested a full tablet report, sending...
I20260810 18:06:39.209322 28579 ts_manager.cc:194] Registered new tserver with Master: 66587d6e1a0248afa176aa20b8989813 (127.21.182.67:42453)
I20260810 18:06:39.209992 28579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:57821
I20260810 18:06:39.210258 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/test-admin.keytab.
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[28525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:39.229282 22233 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 18:06:39.232580 29094 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:39.230767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.21.182.65:34241)
0810 18:06:39.231136 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:39.231153 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:39.231180 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:06:39.231284 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:39.231290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:39.231496 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:39.231621 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:39.232366 (+ 745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33578: BlockingRecv error: recv got EOF from 127.0.0.1:33578 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":141,"threads_started":1}
W20260810 18:06:39.232978 22233 init.cc:244] Error while opening credential cache '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/krb5cc)
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[28525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:39.234233 22233 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[28525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:39.235311 22233 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[28525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.65@KRBTEST.COM
I20260810 18:06:39.246464 29094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:39.236351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.21.182.65:34241)
0810 18:06:39.236509 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:39.236514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:39.236632 (+ 118us) server_negotiation.cc:407] Connection header received
0810 18:06:39.236835 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:39.236838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:39.236883 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:39.236983 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:39.237609 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:39.238499 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:39.239476 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:39.239947 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:39.240078 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:39.242586 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:39.242596 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:39.242611 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:39.242637 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:39.244384 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:39.244890 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:39.244896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:39.244897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:39.244983 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:39.245311 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:39.245315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:39.245317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:39.245628 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:39.245868 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:39.246133 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 18:06:39.246309 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 18:06:39.250875 29094 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:39.249690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.21.182.65:34241)
0810 18:06:39.249915 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:39.249921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:39.249936 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:06:39.250092 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:39.250097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:39.250144 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:39.250236 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:39.250727 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33592: BlockingRecv error: recv got EOF from 127.0.0.1:33592 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
W20260810 18:06:39.251127 22233 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/krb5cc' of type FILE
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[28525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:39.252409 22233 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[28525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:39.253449 22233 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[28525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.65@KRBTEST.COM
I20260810 18:06:39.262400 29094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:39.254334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33608 (local address 127.21.182.65:34241)
0810 18:06:39.254452 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:39.254455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:39.254468 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:06:39.254580 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:39.254582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:39.254618 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:39.254683 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:39.255251 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:39.255758 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:39.256683 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:39.256915 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:39.256996 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:39.259328 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:39.259331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:39.259333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:39.259356 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:39.260737 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:39.261155 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:39.261158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:39.261159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:39.261233 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:39.261471 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:39.261473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:39.261474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:39.261654 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:39.261799 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:39.262077 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 18:06:39.262182 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 18:06:39.265743 29094 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:39.264591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.21.182.65:34241)
0810 18:06:39.264812 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:39.264820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:39.264836 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:06:39.265016 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:39.265022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:39.265067 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:39.265156 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:39.265606 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33614: BlockingRecv error: recv got EOF from 127.0.0.1:33614 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 18:06:39.265874 22233 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786385164390354-22233-0/krb5kdc/krb5cc' of type FILE
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[28525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:39.266911 22233 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[28525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:39.267822 22233 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[28525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.65@KRBTEST.COM
I20260810 18:06:39.276788 29094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:39.268719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33630 (local address 127.21.182.65:34241)
0810 18:06:39.268859 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:39.268869 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:39.269040 (+ 171us) server_negotiation.cc:407] Connection header received
0810 18:06:39.269186 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:39.269189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:39.269234 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:39.269314 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:39.269831 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:39.270324 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:39.271121 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:39.271339 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:39.271452 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:39.273778 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:39.273782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:39.273786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:39.273815 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:39.275120 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:39.275544 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:39.275548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:39.275550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:39.275634 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:39.275953 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:39.275956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:39.275958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:39.276130 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:39.276273 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:39.276516 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 18:06:39.276625 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:06:39.277832 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 28648
I20260810 18:06:39.288015 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 28796
I20260810 18:06:39.302258 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 28945
I20260810 18:06:39.323160 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 28539
2026-08-10T18:06:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2445 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[29110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[29110](info): set up 2 sockets
Aug 10 18:06:39 dist-test-slave-27sr krb5kdc[29110](info): commencing operation
krb5kdc: starting...
W20260810 18:06:40.515578 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:06:40 dist-test-slave-27sr krb5kdc[29110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:06:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.126.keytab.
Entry for principal oryx/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.126.keytab.
I20260810 18:06:40.656919 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:37161
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.126.keytab
--principal=oryx/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:37161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:40.776964 29124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:40.777295 29124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:40.777374 29124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:40.781419 29124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:40.781517 29124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:40.781558 29124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:40.781618 29124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:40.781659 29124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:40.786998 29124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:37161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.126.keytab
--principal=oryx/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:37161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:40.788192 29124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:40.789310 29124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:40.798591 29129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:40.798592 29132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:40.798592 29130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:40.798962 29124 server_base.cc:1034] running on GCE node
I20260810 18:06:40.799757 29124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:40.801011 29124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:40.802218 29124 hybrid_clock.cc:648] HybridClock initialized: now 1786385200802186 us; error 42 us; skew 500 ppm
Aug 10 18:06:40 dist-test-slave-27sr krb5kdc[29110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385200, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:40.805359 29124 init.cc:377] Logged in from keytab as oryx/127.21.182.126@KRBTEST.COM (short username oryx)
I20260810 18:06:40.806305 29124 webserver.cc:460] Webserver started at http://127.21.182.126:36709/ using document root <none> and password file <none>
I20260810 18:06:40.806859 29124 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:40.806917 29124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:40.807087 29124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:40.809276 29124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "f3ff08bd6f294102b42e2970087bcad4"
format_stamp: "Formatted at 2026-08-10 18:06:40 on dist-test-slave-27sr"
server_key: "62a8165a4ba72e44e41f11b378113b39"
server_key_iv: "dc8cc06f9d41aa4f7d3e302618d6d997"
server_key_version: "encryptionkey@0"
I20260810 18:06:40.810081 29124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "f3ff08bd6f294102b42e2970087bcad4"
format_stamp: "Formatted at 2026-08-10 18:06:40 on dist-test-slave-27sr"
server_key: "62a8165a4ba72e44e41f11b378113b39"
server_key_iv: "dc8cc06f9d41aa4f7d3e302618d6d997"
server_key_version: "encryptionkey@0"
I20260810 18:06:40.814359 29124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:06:40.817338 29139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:40.819133 29124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:06:40.819317 29124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "f3ff08bd6f294102b42e2970087bcad4"
format_stamp: "Formatted at 2026-08-10 18:06:40 on dist-test-slave-27sr"
server_key: "62a8165a4ba72e44e41f11b378113b39"
server_key_iv: "dc8cc06f9d41aa4f7d3e302618d6d997"
server_key_version: "encryptionkey@0"
I20260810 18:06:40.819459 29124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:40.846140 29124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:40.850008 29124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:40.850252 29124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:40.858804 29124 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:37161
I20260810 18:06:40.859102 29197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:37161 every 8 connection(s)
I20260810 18:06:40.860076 29124 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:40.863111 29198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:40.864840 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 29124
I20260810 18:06:40.865128 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:40.865455 22233 external_mini_cluster.cc:1428] Setting key 48823c70618d046ece353b99523b1113
I20260810 18:06:40.870611 29198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4: Bootstrap starting.
I20260810 18:06:40.873448 29198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:40.874435 29198 log.cc:824] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:06:40 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
I20260810 18:06:40.876958 29198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4: No bootstrap required, opened a new log
I20260810 18:06:40.881907 29198 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } }
I20260810 18:06:40.882195 29198 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:40.882257 29198 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ff08bd6f294102b42e2970087bcad4, State: Initialized, Role: FOLLOWER
I20260810 18:06:40.882438 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:40.867223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59390 (local address 127.21.182.126:37161)
0810 18:06:40.867704 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:40.867726 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:40.867770 (+ 44us) server_negotiation.cc:407] Connection header received
0810 18:06:40.868656 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:40.868684 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:40.869141 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:40.869557 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:40.870353 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:40.871850 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:40.872876 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:40.873601 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:40.873787 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:40.876848 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:40.876855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:40.876874 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:40.876906 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:40.878985 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:40.879475 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:40.879483 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:40.879491 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:40.879655 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:40.880018 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:40.880021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:40.880023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:40.880575 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:40.880785 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:40.881044 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 18:06:40.881403 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":128,"threads_started":1}
I20260810 18:06:40.882781 29198 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } }
I20260810 18:06:40.882967 29198 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:40.883037 29198 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:40.883136 29198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:40.884262 29198 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } }
I20260810 18:06:40.884636 29198 leader_election.cc:302] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3ff08bd6f294102b42e2970087bcad4; no voters:
I20260810 18:06:40.885010 29198 leader_election.cc:288] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:40.885150 29203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:40.885398 29203 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [term 1 LEADER]: Becoming Leader. State: Replica: f3ff08bd6f294102b42e2970087bcad4, State: Running, Role: LEADER
I20260810 18:06:40.885799 29203 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } }
I20260810 18:06:40.886389 29198 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:40.887748 29205 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3ff08bd6f294102b42e2970087bcad4. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } }
I20260810 18:06:40.887935 29205 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:40.887749 29204 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } }
I20260810 18:06:40.888044 29204 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: This master's current role is: LEADER
W20260810 18:06:40.891410 29219 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:06:40.891486 29219 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:06:40.891814 29214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:40.892557 29214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:40.898444 29214 catalog_manager.cc:1347] Generated new cluster ID: ca89d301d2e14245aea25dc5cf52a4a8
I20260810 18:06:40.898533 29214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:40.924394 29214 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:40.925424 29214 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:40.933384 29214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4: Generated new TSK 0
I20260810 18:06:40.934387 29214 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.65.keytab.
Entry for principal oryx/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.65.keytab.
I20260810 18:06:40.975581 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37161
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.65.keytab
--principal=oryx/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:41.128867 29226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:41.129189 29226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:41.129240 29226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:41.133628 29226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:41.133704 29226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:41.133764 29226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:41.139048 29226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.65.keytab
--principal=oryx/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:41.140146 29226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:41.141170 29226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:41.151638 29234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:41.151619 29231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:41.151623 29232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:41.152083 29226 server_base.cc:1034] running on GCE node
I20260810 18:06:41.152546 29226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:41.153216 29226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:41.154433 29226 hybrid_clock.cc:648] HybridClock initialized: now 1786385201154395 us; error 52 us; skew 500 ppm
Aug 10 18:06:41 dist-test-slave-27sr krb5kdc[29110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385201, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:41.157703 29226 init.cc:377] Logged in from keytab as oryx/127.21.182.65@KRBTEST.COM (short username oryx)
I20260810 18:06:41.158669 29226 webserver.cc:460] Webserver started at http://127.21.182.65:42377/ using document root <none> and password file <none>
I20260810 18:06:41.159276 29226 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:41.159338 29226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:41.159566 29226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:41.162145 29226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "b9f6e36bb9254c95baa2cc33af78a1fe"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "e22e4e9d317aef326fc9a086fda8db96"
server_key_iv: "1e07e4c58c8d6b886d89e93acc4ed1b0"
server_key_version: "encryptionkey@0"
I20260810 18:06:41.162883 29226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "b9f6e36bb9254c95baa2cc33af78a1fe"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "e22e4e9d317aef326fc9a086fda8db96"
server_key_iv: "1e07e4c58c8d6b886d89e93acc4ed1b0"
server_key_version: "encryptionkey@0"
I20260810 18:06:41.167201 29226 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 18:06:41.170167 29242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:41.171510 29226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:41.171676 29226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "b9f6e36bb9254c95baa2cc33af78a1fe"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "e22e4e9d317aef326fc9a086fda8db96"
server_key_iv: "1e07e4c58c8d6b886d89e93acc4ed1b0"
server_key_version: "encryptionkey@0"
I20260810 18:06:41.171804 29226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:41.198669 29226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:41.201963 29226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:41.202172 29226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:41.202788 29226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:41.203684 29226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:41.203754 29226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:41.203821 29226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:41.203869 29226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:41.213625 29226 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:42177
I20260810 18:06:41.213642 29371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:42177 every 8 connection(s)
I20260810 18:06:41.214671 29226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:41.224658 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 29226
I20260810 18:06:41.224776 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:06:41.225138 22233 external_mini_cluster.cc:1428] Setting key c80464b71b50c51845e38aacd782f1bc
Aug 10 18:06:41 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385201, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
I20260810 18:06:41.230612 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:41.216654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:60619 (local address 127.21.182.126:37161)
0810 18:06:41.216831 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:41.216836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:41.217872 (+ 1036us) server_negotiation.cc:407] Connection header received
0810 18:06:41.219629 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:41.219635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:41.219707 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:41.219810 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:41.221467 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:41.222032 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:41.223331 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:41.223578 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:41.223663 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:41.226743 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:41.226746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:41.226748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:41.226771 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:41.228372 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:41.228938 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:41.228948 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:41.228951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:41.229030 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:41.229385 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:41.229390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:41.229392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:41.229586 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:41.229700 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:41.230290 (+ 590us) server_negotiation.cc:299] Negotiation successful
0810 18:06:41.230440 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:06:41.231519 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:41.217068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.65:60619)
0810 18:06:41.217608 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:41.217742 (+ 134us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:41.219320 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:41.220133 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:41.220149 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:41.220708 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:41.221293 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:41.221310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:41.222208 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:41.222211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:41.223219 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:41.223229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:41.223822 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:41.223827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:41.224030 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:41.224764 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:41.224789 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:41.226563 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:41.228503 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:41.228511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:41.228530 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:41.228826 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:41.229126 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:41.229130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:41.229132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:41.229287 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:41.229709 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:41.229719 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:41.230039 (+ 320us) client_negotiation.cc:769] Sending connection context
0810 18:06:41.230245 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 18:06:41.230626 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":127,"threads_started":1}
I20260810 18:06:41.233084 29372 heartbeater.cc:347] Connected to a master server at 127.21.182.126:37161
I20260810 18:06:41.233363 29372 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:41.233893 29372 heartbeater.cc:511] Master 127.21.182.126:37161 requested a full tablet report, sending...
I20260810 18:06:41.235538 29159 ts_manager.cc:194] Registered new tserver with Master: b9f6e36bb9254c95baa2cc33af78a1fe (127.21.182.65:42177)
WARNING: no policy specified for oryx/127.21.182.66@KRBTEST.COM; defaulting to no policy
I20260810 18:06:41.236809 29159 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:60619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.66.keytab.
Entry for principal oryx/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.66.keytab.
I20260810 18:06:41.255311 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37161
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.66.keytab
--principal=oryx/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:41.382771 29378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:41.383085 29378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:41.383129 29378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:41.387610 29378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:41.387693 29378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:41.387759 29378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:41.393630 29378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.66.keytab
--principal=oryx/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:41.394790 29378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:41.395856 29378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:06:41.406708 29378 server_base.cc:1034] running on GCE node
W20260810 18:06:41.406642 29384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:41.406648 29385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:41.406656 29389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:41.407454 29378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:41.408314 29378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:41.409540 29378 hybrid_clock.cc:648] HybridClock initialized: now 1786385201409506 us; error 43 us; skew 500 ppm
Aug 10 18:06:41 dist-test-slave-27sr krb5kdc[29110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385201, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:41.412882 29378 init.cc:377] Logged in from keytab as oryx/127.21.182.66@KRBTEST.COM (short username oryx)
I20260810 18:06:41.413914 29378 webserver.cc:460] Webserver started at http://127.21.182.66:42321/ using document root <none> and password file <none>
I20260810 18:06:41.414553 29378 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:41.414623 29378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:41.414837 29378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:41.417457 29378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "aba4923d0f9b498289e62debae97813e"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "76035a124a1180bd0f95b04a32c805d5"
server_key_iv: "cb4137fe0e794fb2c095f287c38c02ba"
server_key_version: "encryptionkey@0"
I20260810 18:06:41.418143 29378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "aba4923d0f9b498289e62debae97813e"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "76035a124a1180bd0f95b04a32c805d5"
server_key_iv: "cb4137fe0e794fb2c095f287c38c02ba"
server_key_version: "encryptionkey@0"
I20260810 18:06:41.422922 29378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 18:06:41.426368 29396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:41.428185 29378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 18:06:41.428340 29378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "aba4923d0f9b498289e62debae97813e"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "76035a124a1180bd0f95b04a32c805d5"
server_key_iv: "cb4137fe0e794fb2c095f287c38c02ba"
server_key_version: "encryptionkey@0"
I20260810 18:06:41.428519 29378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:41.450203 29378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:41.454867 29378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:41.455130 29378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:41.456039 29378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:41.457077 29378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:41.457134 29378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:41.457181 29378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:41.457239 29378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:41.467760 29378 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:36975
I20260810 18:06:41.467796 29521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:36975 every 8 connection(s)
I20260810 18:06:41.468899 29378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:06:41.474547 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 29378
I20260810 18:06:41.474707 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:06:41.475003 22233 external_mini_cluster.cc:1428] Setting key 5c297038603baa9725bf9a6018e22fff
Aug 10 18:06:41 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385201, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
I20260810 18:06:41.485369 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:41.471067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:46681 (local address 127.21.182.126:37161)
0810 18:06:41.471233 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:41.471237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:41.472257 (+ 1020us) server_negotiation.cc:407] Connection header received
0810 18:06:41.473742 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:41.473746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:41.473815 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:41.473920 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:41.475329 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:41.475984 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:41.477341 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:41.477761 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:41.477895 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:41.480925 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:41.480929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:41.480931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:41.480958 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:41.482672 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:41.483440 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:41.483444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:41.483446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:41.483539 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:41.484050 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:41.484054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:41.484058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:41.484282 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:41.484404 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:41.485022 (+ 618us) server_negotiation.cc:299] Negotiation successful
0810 18:06:41.485178 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:06:41.486445 29524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:41.471438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.66:46681)
0810 18:06:41.472062 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:41.472110 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:41.473495 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:41.474114 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:41.474122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:41.474525 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:41.475152 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:41.475168 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:41.476116 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:41.476120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:41.477153 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:41.477163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:41.477934 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:41.477940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:41.478165 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:41.478934 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:41.478962 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:41.480718 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:41.482917 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:41.482925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:41.482946 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:41.483296 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:41.483670 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:41.483674 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:41.483676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:41.483864 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:41.484404 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:41.484413 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:41.484730 (+ 317us) client_negotiation.cc:769] Sending connection context
0810 18:06:41.484952 (+ 222us) client_negotiation.cc:240] Negotiation successful
0810 18:06:41.485368 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for oryx/127.21.182.67@KRBTEST.COM; defaulting to no policy
I20260810 18:06:41.488067 29522 heartbeater.cc:347] Connected to a master server at 127.21.182.126:37161
I20260810 18:06:41.488385 29522 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:41.489156 29522 heartbeater.cc:511] Master 127.21.182.126:37161 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.182.67@KRBTEST.COM" created.
I20260810 18:06:41.490455 29159 ts_manager.cc:194] Registered new tserver with Master: aba4923d0f9b498289e62debae97813e (127.21.182.66:36975)
I20260810 18:06:41.491212 29159 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:46681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.67.keytab.
Entry for principal oryx/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.67.keytab.
I20260810 18:06:41.512847 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37161
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.67.keytab
--principal=oryx/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:41.655503 29527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:41.655844 29527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:41.655958 29527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:41.660617 29527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:41.660704 29527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:41.660818 29527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:41.666712 29527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.67.keytab
--principal=oryx/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:41.668123 29527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:41.669340 29527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:41.680395 29534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:41.680397 29537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:41.680675 29527 server_base.cc:1034] running on GCE node
W20260810 18:06:41.680397 29535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:41.681335 29527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:41.682157 29527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:41.683390 29527 hybrid_clock.cc:648] HybridClock initialized: now 1786385201683352 us; error 52 us; skew 500 ppm
Aug 10 18:06:41 dist-test-slave-27sr krb5kdc[29110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385201, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:41.686746 29527 init.cc:377] Logged in from keytab as oryx/127.21.182.67@KRBTEST.COM (short username oryx)
I20260810 18:06:41.687867 29527 webserver.cc:460] Webserver started at http://127.21.182.67:42605/ using document root <none> and password file <none>
I20260810 18:06:41.688602 29527 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:41.688670 29527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:41.688863 29527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:41.691035 29527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "28ec792c34e74d42a3c29be691aac1ca"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "83bd121de3600f4d900f5e38b47c4bac"
server_key_iv: "d8fabd780212820eff0a03e888a103f2"
server_key_version: "encryptionkey@0"
I20260810 18:06:41.691658 29527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "28ec792c34e74d42a3c29be691aac1ca"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "83bd121de3600f4d900f5e38b47c4bac"
server_key_iv: "d8fabd780212820eff0a03e888a103f2"
server_key_version: "encryptionkey@0"
I20260810 18:06:41.696177 29527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 18:06:41.699762 29546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:41.701284 29527 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 18:06:41.701452 29527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "28ec792c34e74d42a3c29be691aac1ca"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "83bd121de3600f4d900f5e38b47c4bac"
server_key_iv: "d8fabd780212820eff0a03e888a103f2"
server_key_version: "encryptionkey@0"
I20260810 18:06:41.701617 29527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:41.728845 29527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:41.733053 29527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:41.733311 29527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:41.734138 29527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:41.735165 29527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:41.735249 29527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:41.735328 29527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:41.735378 29527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:41.746117 29527 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:40943
I20260810 18:06:41.746140 29672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:40943 every 8 connection(s)
I20260810 18:06:41.747272 29527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:06:41.752926 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 29527
I20260810 18:06:41.753057 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:06:41.753371 22233 external_mini_cluster.cc:1428] Setting key a9973837c94a2567ba2574129e566186
Aug 10 18:06:41 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385201, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
I20260810 18:06:41.763618 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:41.749485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:53605 (local address 127.21.182.126:37161)
0810 18:06:41.749690 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:41.749696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:41.750771 (+ 1075us) server_negotiation.cc:407] Connection header received
0810 18:06:41.752321 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:41.752325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:41.752387 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:41.752478 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:41.753981 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:41.754797 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:41.756432 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:41.756669 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:41.756794 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:41.759635 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:41.759638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:41.759641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:41.759674 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:41.761279 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:41.761857 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:41.761861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:41.761863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:41.761966 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:41.762353 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:41.762356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:41.762358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:41.762570 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:41.762725 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:41.763300 (+ 575us) server_negotiation.cc:299] Negotiation successful
0810 18:06:41.763441 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 18:06:41.764619 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:41.749993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.67:53605)
0810 18:06:41.750552 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:41.750603 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:41.752035 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:41.752681 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:41.752689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:41.753197 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:41.753811 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:41.753827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:41.754981 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:41.754985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:41.756294 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:41.756305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:41.756857 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:41.756861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:41.757046 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:41.757733 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:41.757752 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:41.759430 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:41.761413 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:41.761420 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:41.761439 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:41.761743 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:41.762073 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:41.762076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:41.762079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:41.762247 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:41.762725 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:41.762733 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:41.763027 (+ 294us) client_negotiation.cc:769] Sending connection context
0810 18:06:41.763251 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 18:06:41.763643 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":146,"threads_started":1}
I20260810 18:06:41.766043 29673 heartbeater.cc:347] Connected to a master server at 127.21.182.126:37161
I20260810 18:06:41.766321 29673 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:41.766836 29673 heartbeater.cc:511] Master 127.21.182.126:37161 requested a full tablet report, sending...
I20260810 18:06:41.768174 29159 ts_manager.cc:194] Registered new tserver with Master: 28ec792c34e74d42a3c29be691aac1ca (127.21.182.67:40943)
I20260810 18:06:41.768889 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:06:41.769075 29159 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:53605
WARNING: no policy specified for oryx/127.21.182.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.182.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.182.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.125.keytab.
Entry for principal oryx/127.21.182.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.125.keytab.
I20260810 18:06:41.801441 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.125:35871
--webserver_interface=127.21.182.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.125.keytab
--principal=oryx/127.21.182.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:37161,127.21.182.125:35871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:41.927373 29679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:41.927736 29679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:41.927845 29679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:41.932400 29679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:41.932500 29679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:41.932539 29679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:41.932595 29679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:41.932631 29679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:41.938514 29679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:37161,127.21.182.125:35871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.125.keytab
--principal=oryx/127.21.182.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.125:35871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.182.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
W20260810 18:06:41.938846 29679 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.182.126:37161,127.21.182.125:35871'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 18:06:41.939841 29679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:41.940999 29679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:41.950520 29685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:41.950495 29684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:41.950495 29687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:41.950557 29679 server_base.cc:1034] running on GCE node
I20260810 18:06:41.951475 29679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:41.952826 29679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:41.954046 29679 hybrid_clock.cc:648] HybridClock initialized: now 1786385201954008 us; error 57 us; skew 500 ppm
Aug 10 18:06:41 dist-test-slave-27sr krb5kdc[29110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385201, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:41.957423 29679 init.cc:377] Logged in from keytab as oryx/127.21.182.125@KRBTEST.COM (short username oryx)
I20260810 18:06:41.958514 29679 webserver.cc:460] Webserver started at http://127.21.182.125:45067/ using document root <none> and password file <none>
I20260810 18:06:41.959138 29679 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:41.959262 29679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:41.959511 29679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:41.961954 29679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/data/instance:
uuid: "31a294944c674d20ac6295f3ebbd2f9d"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "c77d924502fa754213daa9a37851ddd4"
server_key_iv: "a6646a951b843e3532d6d8b357da191a"
server_key_version: "encryptionkey@0"
I20260810 18:06:41.962604 29679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/wal/instance:
uuid: "31a294944c674d20ac6295f3ebbd2f9d"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "c77d924502fa754213daa9a37851ddd4"
server_key_iv: "a6646a951b843e3532d6d8b357da191a"
server_key_version: "encryptionkey@0"
I20260810 18:06:41.967017 29679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 18:06:41.969969 29694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:41.971388 29679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:41.971544 29679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/wal
uuid: "31a294944c674d20ac6295f3ebbd2f9d"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "c77d924502fa754213daa9a37851ddd4"
server_key_iv: "a6646a951b843e3532d6d8b357da191a"
server_key_version: "encryptionkey@0"
I20260810 18:06:41.971669 29679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:41.983728 29679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:41.987403 29679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:41.987654 29679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:41.995553 29679 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.125:35871
I20260810 18:06:41.995867 29749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.125:35871 every 8 connection(s)
I20260810 18:06:41.997030 29679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/data/info.pb
I20260810 18:06:42.000296 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 29679
I20260810 18:06:42.000557 29750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:42.006057 29750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.182.125@KRBTEST.COM
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385201, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.125@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
I20260810 18:06:42.018965 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.002103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.21.182.125:35871)
0810 18:06:42.002648 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.002665 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.002714 (+ 49us) server_negotiation.cc:407] Connection header received
0810 18:06:42.003804 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.003827 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.004326 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.004729 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.005711 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.006994 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.008142 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.008696 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.008894 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.012137 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.012145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.012162 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.012208 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.014542 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.015173 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.015180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.015188 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.015349 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.015818 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.015842 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.015845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.016545 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.016806 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.017300 (+ 494us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.017715 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":175,"threads_started":1}
I20260810 18:06:42.019323 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.008053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.21.182.126:37161)
0810 18:06:42.008914 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.008918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.008931 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:06:42.008972 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.008974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.009015 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.009120 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.009873 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.010394 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.011787 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.012127 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.012253 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.015147 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.015151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.015154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.015188 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.017134 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.017714 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.017717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.017719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.017832 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.018211 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.018214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.018216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.018534 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.018655 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.019021 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.019161 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774}
I20260810 18:06:42.019374 29755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.008048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:33914)
0810 18:06:42.008470 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.008524 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.008650 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.009178 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.009185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.009585 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.009727 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.009733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.010541 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.010544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.011643 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.011654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.012260 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.012264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.012359 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.013107 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.013119 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.014946 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.017276 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.017287 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.017294 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.017599 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.017999 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.018002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.018003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.018091 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.018671 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.018677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.018801 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.018929 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.019157 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":145,"threads_started":1}
I20260810 18:06:42.021958 29750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385201, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.125@KRBTEST.COM for oryx/127.21.182.125@KRBTEST.COM
I20260810 18:06:42.031188 29755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.022559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:35871 (local address 127.0.0.1:38390)
0810 18:06:42.022719 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.022733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.022841 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.023080 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.023083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.023291 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.023384 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.023390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.024681 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.024689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.025269 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.025279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.025759 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.025762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.025855 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.026347 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.026359 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.027824 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.029622 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.029628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.029631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.029837 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.030119 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.030123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.030126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.030210 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.030619 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.030622 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.030720 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.030834 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.030987 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 18:06:42.031188 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.022725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.21.182.125:35871)
0810 18:06:42.022829 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.022833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.022846 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:06:42.022936 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.022939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.022980 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.023068 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.023506 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.024532 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.025384 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.025610 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.025750 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.028021 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.028024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.028026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.028047 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.029504 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.029939 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.029944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.029946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.030024 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.030310 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.030314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.030315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.030512 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.030622 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.030882 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.031044 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":46}
I20260810 18:06:42.034904 29750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: Bootstrap starting.
I20260810 18:06:42.037492 29750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:42.038360 29750 log.cc:824] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:42.040591 29750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: No bootstrap required, opened a new log
I20260810 18:06:42.043689 29750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } }
I20260810 18:06:42.043969 29750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:42.044034 29750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a294944c674d20ac6295f3ebbd2f9d, State: Initialized, Role: FOLLOWER
I20260810 18:06:42.044466 29750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } }
I20260810 18:06:42.044978 29757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } } }
I20260810 18:06:42.045104 29757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:06:42.045796 29750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: configured and running, proceeding with master startup.
W20260810 18:06:42.051204 29768 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:06:42.051326 29768 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:06:42.054010 29679 master_runner.cc:428] Detected that this master 31a294944c674d20ac6295f3ebbd2f9d is joining an existing cluster
I20260810 18:06:42.054106 29679 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.182.125:35871
I20260810 18:06:42.059444 29757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:42.059700 29757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } }
I20260810 18:06:42.060766 29757 leader_election.cc:288] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3ff08bd6f294102b42e2970087bcad4 (127.21.182.126:37161)
I20260810 18:06:42.061637 29169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31a294944c674d20ac6295f3ebbd2f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ff08bd6f294102b42e2970087bcad4" is_pre_election: true
I20260810 18:06:42.062842 29695 leader_election.cc:302] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 31a294944c674d20ac6295f3ebbd2f9d; no voters: f3ff08bd6f294102b42e2970087bcad4
I20260810 18:06:42.063205 29757 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:06:42.067031 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.056985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.21.182.125:35871)
0810 18:06:42.057123 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.057127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.057793 (+ 666us) server_negotiation.cc:407] Connection header received
0810 18:06:42.057885 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.057888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.057934 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.058001 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.058695 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.059451 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.060569 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.061133 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.061261 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.063415 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.063419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.063421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.063459 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.065240 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.065706 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.065710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.065712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.065787 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.066091 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.066095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.066097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.066340 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.066450 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.066720 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.066883 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":58,"spinlock_wait_cycles":6400}
I20260810 18:06:42.067045 29775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.057364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:35871 (local address 127.0.0.1:38396)
0810 18:06:42.057686 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.057701 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.057802 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.058119 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.058122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.058413 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.058554 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.058561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.059577 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.059582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.060441 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.060451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.061384 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.061387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.061478 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.062739 (+ 1261us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.062755 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.063277 (+ 522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.065361 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.065364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.065366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.065611 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.065881 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.065883 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.065886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.065989 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.066439 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.066442 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.066542 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.066651 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.066826 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":125,"threads_started":1}
I20260810 18:06:42.067632 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.057048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33922 (local address 127.21.182.126:37161)
0810 18:06:42.057257 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.057262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.057510 (+ 248us) server_negotiation.cc:407] Connection header received
0810 18:06:42.057739 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.057745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.057810 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.057894 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.058524 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.059109 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.061596 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.062039 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.062123 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.064174 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.064177 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.064179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.064207 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.065765 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.066261 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.066265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.066267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.066358 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.066679 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.066683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.066684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.066903 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.066974 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.067349 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.067476 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260810 18:06:42.068210 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.057038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:33922)
0810 18:06:42.057411 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.057428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.057597 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.057922 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.057925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.058263 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.058390 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.058396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.059297 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.059300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.061441 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.061453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.062304 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.062308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.062403 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.063466 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.063481 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.064016 (+ 535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.065884 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.065890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.065892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.066144 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.066471 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.066475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.066477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.066574 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.067048 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.067051 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.067153 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.067849 (+ 696us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.068031 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"spinlock_wait_cycles":1536,"thread_start_us":155,"threads_started":1}
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385200, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.126@KRBTEST.COM for oryx/127.21.182.125@KRBTEST.COM
WARNING: no policy specified for oryx/127.21.182.124@KRBTEST.COM; defaulting to no policy
I20260810 18:06:42.085301 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.074440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.21.182.125:35871)
0810 18:06:42.074551 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.074555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.074568 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:06:42.074639 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.074642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.074693 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.074780 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.075847 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.076551 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.077571 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.077802 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.077926 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.080659 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.080667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.080672 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.080712 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.082668 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.083398 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.083403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.083405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.083503 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.083861 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.083866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.083869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.084259 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.084365 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.084919 (+ 554us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.085074 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 18:06:42.086048 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.073625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:35871 (local address 127.0.0.1:38402)
0810 18:06:42.074115 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.074183 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.074332 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.074847 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.074857 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.075463 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.075663 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.075675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.076719 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.076722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.077439 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.077448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.077938 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.077941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.078073 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.078726 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.078744 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.080421 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.082835 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.082852 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.082861 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.083258 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.083634 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.083638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.083641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.083743 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.084474 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.084485 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.084654 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.085534 (+ 880us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.085779 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.182.124@KRBTEST.COM" created.
I20260810 18:06:42.088091 29159 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.182.125:35871: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d"
member_type: NON_VOTER
last_known_addr {
host: "127.21.182.125"
port: 35871
}
attrs {
promote: true
}
}
dest_uuid: "f3ff08bd6f294102b42e2970087bcad4"
cas_config_opid_index: -1
I20260810 18:06:42.089183 29159 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: NON_VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: true } }
I20260810 18:06:42.092202 29724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:42.093703 29724 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 FOLLOWER]: Refusing update from remote peer f3ff08bd6f294102b42e2970087bcad4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 18:06:42.091408 29783 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 31a294944c674d20ac6295f3ebbd2f9d (127.21.182.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: NON_VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: true } } }
I20260810 18:06:42.094640 29782 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: NON_VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:42.095662 29783 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.182.125:35871
I20260810 18:06:42.096122 29783 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: NON_VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: true } } }
I20260810 18:06:42.096254 29783 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:42.096347 29679 master.cc:537] Master@127.21.182.125:35871 shutting down...
W20260810 18:06:42.098184 29143 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 18:06:42.099834 29143 consensus_peers.cc:487] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 -> Peer 31a294944c674d20ac6295f3ebbd2f9d (127.21.182.125:35871): Couldn't send request to peer 31a294944c674d20ac6295f3ebbd2f9d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:06:42.102847 29679 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:42.103230 29679 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:42.103381 29679 tablet_replica.cc:330] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.182.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.124.keytab.
Entry for principal oryx/127.21.182.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.124.keytab.
I20260810 18:06:42.116271 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.124:46311
--webserver_interface=127.21.182.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.124.keytab
--principal=oryx/127.21.182.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:37161,127.21.182.125:35871,127.21.182.124:46311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:06:42.119179 29679 master.cc:552] Master@127.21.182.125:35871 shutdown complete.
I20260810 18:06:42.119266 29679 master_runner.cc:305] Clearing existing system tablet
I20260810 18:06:42.119549 29679 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:42.119613 29679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:42.121189 29679 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:06:42.122287 29792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:42.122602 29679 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:42.122753 29679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:06:42.122789 29679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/wal
uuid: "31a294944c674d20ac6295f3ebbd2f9d"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "c77d924502fa754213daa9a37851ddd4"
server_key_iv: "a6646a951b843e3532d6d8b357da191a"
server_key_version: "encryptionkey@0"
I20260810 18:06:42.125438 29679 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:42.128082 29679 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 18:06:42.128193 29679 log.cc:1196] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 18:06:42.128480 29679 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: Deleting consensus metadata
I20260810 18:06:42.128677 29679 master_runner.cc:315] Copying system tablet from 127.21.182.126:37161
I20260810 18:06:42.130082 29679 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: tablet copy: Beginning tablet copy session from remote peer at address 127.21.182.126:37161
I20260810 18:06:42.140467 29799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.130796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:33938)
0810 18:06:42.131236 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.131253 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.131394 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.131738 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.131741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.132009 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.132161 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.132169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.133338 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.133342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.134342 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.134354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.134989 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.134993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.135096 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.135878 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.136005 (+ 127us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.136477 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.138635 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.138652 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.138654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.138931 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.139256 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.139259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.139262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.139362 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.139824 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.139828 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.139965 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.140076 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.140244 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":101,"threads_started":1}
I20260810 18:06:42.140476 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.130912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33938 (local address 127.21.182.126:37161)
0810 18:06:42.131052 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.131057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.131369 (+ 312us) server_negotiation.cc:407] Connection header received
0810 18:06:42.131522 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.131526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.131590 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.131695 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.132316 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.133207 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.134474 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.134852 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.134962 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.136649 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.136653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.136657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.136688 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.138510 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.139040 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.139044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.139046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.139147 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.139463 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.139466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.139468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.139714 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.139822 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.140162 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.140302 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:06:42.141343 29179 tablet_copy_service.cc:140] P f3ff08bd6f294102b42e2970087bcad4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 31a294944c674d20ac6295f3ebbd2f9d ({username='oryx', principal='oryx/127.21.182.125@KRBTEST.COM'} at 127.0.0.1:33938)
I20260810 18:06:42.141516 29179 tablet_copy_service.cc:161] P f3ff08bd6f294102b42e2970087bcad4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 31a294944c674d20ac6295f3ebbd2f9d at {username='oryx', principal='oryx/127.21.182.125@KRBTEST.COM'} at 127.0.0.1:33938: session id = 31a294944c674d20ac6295f3ebbd2f9d-00000000000000000000000000000000
I20260810 18:06:42.143458 29179 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:06:42.145531 29679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 18:06:42.148447 29679 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: tablet copy: Starting download of 0 data blocks...
I20260810 18:06:42.148680 29679 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: tablet copy: Starting download of 1 WAL segments...
I20260810 18:06:42.150833 29679 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:06:42.153103 29179 tablet_copy_service.cc:342] P f3ff08bd6f294102b42e2970087bcad4: Request end of tablet copy session 31a294944c674d20ac6295f3ebbd2f9d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.182.125@KRBTEST.COM'} at 127.0.0.1:33938
I20260810 18:06:42.153240 29179 tablet_copy_service.cc:433] P f3ff08bd6f294102b42e2970087bcad4: ending tablet copy session 31a294944c674d20ac6295f3ebbd2f9d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 31a294944c674d20ac6295f3ebbd2f9d
I20260810 18:06:42.161721 29679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:42.163843 29800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:42.164079 29805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:42.164335 29803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:42.164594 29679 server_base.cc:1034] running on GCE node
I20260810 18:06:42.164749 29679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:42.165091 29679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:42.166244 29679 hybrid_clock.cc:648] HybridClock initialized: now 1786385202166220 us; error 49 us; skew 500 ppm
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385202, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:42.168063 29679 init.cc:377] Logged in from keytab as oryx/127.21.182.125@KRBTEST.COM (short username oryx)
I20260810 18:06:42.168623 29679 webserver.cc:460] Webserver started at http://127.21.182.125:36429/ using document root <none> and password file <none>
I20260810 18:06:42.168845 29679 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:42.168892 29679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:42.170497 29679 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 18:06:42.171799 29813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:42.172356 29679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:06:42.172583 29679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/wal
uuid: "31a294944c674d20ac6295f3ebbd2f9d"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "c77d924502fa754213daa9a37851ddd4"
server_key_iv: "a6646a951b843e3532d6d8b357da191a"
server_key_version: "encryptionkey@0"
I20260810 18:06:42.172739 29679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:42.188328 29679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:42.190847 29679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:42.196200 29679 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.125:35871
I20260810 18:06:42.196250 29870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.125:35871 every 8 connection(s)
I20260810 18:06:42.197265 29679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-1/data/info.pb
I20260810 18:06:42.199465 29871 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:06:42.200592 29871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: Bootstrap starting.
I20260810 18:06:42.212530 29871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:06:42.212936 29871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: Bootstrap complete.
I20260810 18:06:42.213788 29871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: NON_VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: true } }
I20260810 18:06:42.213930 29871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31a294944c674d20ac6295f3ebbd2f9d, State: Initialized, Role: LEARNER
I20260810 18:06:42.214038 29871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: NON_VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: true } }
I20260810 18:06:42.214524 29871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:42.219391 29875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: NON_VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: true } } }
I20260810 18:06:42.219560 29875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: This master's current role is: LEARNER
I20260810 18:06:42.221122 29886 catalog_manager.cc:1259] Loaded cluster ID: ca89d301d2e14245aea25dc5cf52a4a8
I20260810 18:06:42.221194 29886 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: loading cluster ID for follower catalog manager: success
I20260810 18:06:42.223547 29886 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: acquiring CA information for follower catalog manager: success
I20260810 18:06:42.225191 29886 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:06:42.239459 29372 heartbeater.cc:503] Master 127.21.182.126:37161 was elected leader, sending a full tablet report...
W20260810 18:06:42.266350 29790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:42.266665 29790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:42.266708 29790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:42.270922 29790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:42.271004 29790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:42.271024 29790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:42.271041 29790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:42.271056 29790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:42.277226 29790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:37161,127.21.182.125:35871,127.21.182.124:46311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.124.keytab
--principal=oryx/127.21.182.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.124:46311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.182.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:42.278414 29790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:42.279516 29790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:42.288728 29896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:42.288826 29894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:42.289103 29893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:42.289378 29790 server_base.cc:1034] running on GCE node
I20260810 18:06:42.290019 29790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:42.291205 29790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:42.292392 29790 hybrid_clock.cc:648] HybridClock initialized: now 1786385202292370 us; error 72 us; skew 500 ppm
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385202, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:42.295758 29790 init.cc:377] Logged in from keytab as oryx/127.21.182.124@KRBTEST.COM (short username oryx)
I20260810 18:06:42.296859 29790 webserver.cc:460] Webserver started at http://127.21.182.124:33519/ using document root <none> and password file <none>
I20260810 18:06:42.297493 29790 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:42.297593 29790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:42.297864 29790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:42.300272 29790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/data/instance:
uuid: "ea36ca408c3f4a00b71c76b044ca6543"
format_stamp: "Formatted at 2026-08-10 18:06:42 on dist-test-slave-27sr"
server_key: "d2334155366a3ea71b8c80e2c18e64a8"
server_key_iv: "69748db13fe22d6d55d2a5a246fdc7a9"
server_key_version: "encryptionkey@0"
I20260810 18:06:42.300902 29790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/wal/instance:
uuid: "ea36ca408c3f4a00b71c76b044ca6543"
format_stamp: "Formatted at 2026-08-10 18:06:42 on dist-test-slave-27sr"
server_key: "d2334155366a3ea71b8c80e2c18e64a8"
server_key_iv: "69748db13fe22d6d55d2a5a246fdc7a9"
server_key_version: "encryptionkey@0"
I20260810 18:06:42.305272 29790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:06:42.308585 29903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:42.310081 29790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:06:42.310247 29790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/wal
uuid: "ea36ca408c3f4a00b71c76b044ca6543"
format_stamp: "Formatted at 2026-08-10 18:06:42 on dist-test-slave-27sr"
server_key: "d2334155366a3ea71b8c80e2c18e64a8"
server_key_iv: "69748db13fe22d6d55d2a5a246fdc7a9"
server_key_version: "encryptionkey@0"
I20260810 18:06:42.310393 29790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:42.326104 29790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:42.330077 29790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:42.330327 29790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:42.339087 29790 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.124:46311
I20260810 18:06:42.339387 29962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.124:46311 every 8 connection(s)
I20260810 18:06:42.340468 29790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/data/info.pb
I20260810 18:06:42.343828 29963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:42.345888 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 29790
I20260810 18:06:42.349293 29963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.182.124@KRBTEST.COM
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385202, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.124@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
I20260810 18:06:42.363494 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.352103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.21.182.126:37161)
0810 18:06:42.352277 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.352282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.353235 (+ 953us) server_negotiation.cc:407] Connection header received
0810 18:06:42.353379 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.353385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.353449 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.353551 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.354289 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.354880 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.355927 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.356360 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.356513 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.359692 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.359696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.359698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.359725 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.361510 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.362021 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.362025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.362027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.362120 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.362423 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.362426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.362428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.362630 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.362748 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.363145 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.363282 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:06:42.364517 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.348069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.21.182.124:46311)
0810 18:06:42.348749 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.348775 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.348826 (+ 51us) server_negotiation.cc:407] Connection header received
0810 18:06:42.349848 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.349873 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.350332 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.350756 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.351860 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.353257 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.354363 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.354954 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.355148 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.358774 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.358782 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.358799 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.358832 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.360847 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.361409 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.361415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.361422 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.361539 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.361972 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.361977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.361979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.362462 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.362725 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.363014 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.363458 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":423,"thread_start_us":185,"threads_started":1}
I20260810 18:06:42.364530 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.351872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:33954)
0810 18:06:42.353068 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.353119 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.353268 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.353604 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.353609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.353997 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.354133 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.354139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.355021 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.355025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.355763 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.355772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.356561 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.356564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.356654 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.357404 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.357432 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.359518 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.361624 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.361636 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.361644 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.361907 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.362220 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.362223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.362225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.362322 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.362750 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.362760 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.362925 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.363084 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.363558 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"mutex_wait_us":149,"thread_start_us":866,"threads_started":1}
I20260810 18:06:42.366675 29963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385202, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.124@KRBTEST.COM for oryx/127.21.182.125@KRBTEST.COM
I20260810 18:06:42.376435 29967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.367505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.21.182.125:35871)
0810 18:06:42.367800 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.367803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.367815 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:06:42.367853 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.367855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.367943 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.368057 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.368479 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.369205 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.370023 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.370281 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.370408 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.372803 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.372812 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.372814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.372839 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.374541 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.375033 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.375041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.375043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.375144 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.375442 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.375446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.375448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.375678 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.375797 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.376144 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.376276 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":113,"threads_started":1}
I20260810 18:06:42.376433 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.367335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:35871 (local address 127.0.0.1:38416)
0810 18:06:42.367467 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.367481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.367549 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.368073 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.368076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.368254 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.368356 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.368362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.369322 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.369326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.369890 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.369899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.370400 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.370403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.370509 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.371079 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.371092 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.372647 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.374686 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.374689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.374691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.374913 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.375245 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.375248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.375250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.375342 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.375787 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.375790 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.375955 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.376085 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.376244 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 18:06:42.377475 29963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385202, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.124@KRBTEST.COM for oryx/127.21.182.124@KRBTEST.COM
I20260810 18:06:42.386059 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.378103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.21.182.124:46311)
0810 18:06:42.378253 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.378257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.378269 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:06:42.378300 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.378302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.378336 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.378403 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.378807 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.379325 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.380655 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.380932 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.381021 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.383186 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.383189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.383191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.383236 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.384605 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.385007 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.385010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.385011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.385086 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.385328 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.385330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.385331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.385509 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.385585 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.385823 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.385922 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:06:42.386059 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.377992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:46311 (local address 127.0.0.1:43298)
0810 18:06:42.378117 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.378130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.378197 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.378435 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.378437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.378575 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.378671 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.378676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.380023 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.380026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.380536 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.380545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.381042 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.381044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.381104 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.381635 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.381729 (+ 94us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.383044 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.384716 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.384718 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.384720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.384919 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.385169 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.385171 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.385172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.385248 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.385591 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.385593 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.385674 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.385755 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.385861 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 18:06:42.389640 29963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: Bootstrap starting.
I20260810 18:06:42.392241 29963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:42.393059 29963 log.cc:824] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:42.395190 29963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: No bootstrap required, opened a new log
I20260810 18:06:42.398381 29963 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } }
I20260810 18:06:42.398654 29963 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:42.398744 29963 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea36ca408c3f4a00b71c76b044ca6543, State: Initialized, Role: FOLLOWER
I20260810 18:06:42.399236 29963 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } }
I20260810 18:06:42.399824 29969 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } } }
I20260810 18:06:42.400022 29969 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:06:42.400676 29963 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 18:06:42.406065 29981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:06:42.406183 29981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:06:42.409929 29790 master_runner.cc:428] Detected that this master ea36ca408c3f4a00b71c76b044ca6543 is joining an existing cluster
I20260810 18:06:42.410006 29790 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.182.124:46311
I20260810 18:06:42.418532 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 29226
I20260810 18:06:42.425625 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.413559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.21.182.124:46311)
0810 18:06:42.413778 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.413783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.414464 (+ 681us) server_negotiation.cc:407] Connection header received
0810 18:06:42.414618 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.414622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.414691 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.414745 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.415470 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.416307 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.417913 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.418277 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.418369 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.420601 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.420608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.420613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.420641 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.423030 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.423697 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.423703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.423708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.423786 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.424190 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.424197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.424203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.424468 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.424585 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.425056 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.425309 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:06:42.426505 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.413888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:46311 (local address 127.0.0.1:43300)
0810 18:06:42.414373 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.414391 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.414523 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.414900 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.414903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.415136 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.415335 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.415342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.416614 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.416619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.417772 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.417784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.418528 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.418532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.418715 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.419971 (+ 1256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.419991 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.420456 (+ 465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.423203 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.423209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.423215 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.423583 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.423882 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.423885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.423944 (+ 59us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.424042 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.424708 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.424711 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.424842 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.425915 (+ 1073us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.426135 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":92,"thread_start_us":193,"threads_started":1}
I20260810 18:06:42.430308 29988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.412822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:35871 (local address 127.0.0.1:38420)
0810 18:06:42.413413 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.413429 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.413607 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.414024 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.414027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.414330 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.414492 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.414501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.415601 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.415604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.416612 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.416624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.417409 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.417415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.417509 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.418137 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.418155 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.425815 (+ 7660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.428212 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.428217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.428221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.428487 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.428936 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.428948 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.428952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.429058 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.429706 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.429711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.429829 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.429937 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.430110 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":212,"spinlock_wait_cycles":13824,"thread_start_us":111,"threads_started":1}
I20260810 18:06:42.430564 29967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.412978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.21.182.125:35871)
0810 18:06:42.413234 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.413238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.413505 (+ 267us) server_negotiation.cc:407] Connection header received
0810 18:06:42.413729 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.413733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.413794 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.413850 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.414643 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.415452 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.416757 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.417115 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.417198 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.426006 (+ 8808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.426013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.426018 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.426054 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.428084 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.428628 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.428636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.428642 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.428798 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.429179 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.429187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.429192 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.429575 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.429738 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.430054 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.430274 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260810 18:06:42.439146 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.412959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.21.182.126:37161)
0810 18:06:42.413138 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.413144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.413509 (+ 365us) server_negotiation.cc:407] Connection header received
0810 18:06:42.413769 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.413774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.413838 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.413902 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.414832 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.415811 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.417929 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.418315 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.418394 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.434946 (+ 16552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.434951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.434954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.434990 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.436945 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.437591 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.437600 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.437602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.437707 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.438062 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.438065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.438067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.438317 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.438441 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.438823 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.438962 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 18:06:42.439327 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.412822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:33960)
0810 18:06:42.413415 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.413434 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.413613 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.414240 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.414243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.414535 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.414683 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.414690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.416090 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.416093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.417785 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.417793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.418544 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.418547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.433320 (+ 14773us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.434022 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.434068 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.434698 (+ 630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.437116 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.437123 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.437129 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.437457 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.437823 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.437830 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.437835 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.437953 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.438436 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.438443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.438591 (+ 148us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.438743 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.438953 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":33,"spinlock_wait_cycles":12416,"thread_start_us":121,"threads_started":1}
I20260810 18:06:42.440624 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:42177
--local_ip_for_outbound_sockets=127.21.182.65
--tserver_master_addrs=127.21.182.126:37161
--webserver_port=42377
--webserver_interface=127.21.182.65
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.65.keytab
--principal=oryx/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.182.126:37161,127.21.182.125:35871
--tserver_master_addrs=127.21.182.126:37161,127.21.182.125:35871,127.21.182.124:46311 with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385200, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.126@KRBTEST.COM for oryx/127.21.182.124@KRBTEST.COM
I20260810 18:06:42.476727 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.444302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.21.182.124:46311)
0810 18:06:42.444467 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.444475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.444494 (+ 19us) server_negotiation.cc:407] Connection header received
0810 18:06:42.444676 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.444683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.444772 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.444880 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.445406 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.446394 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.447638 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.448149 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.448292 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.450939 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.450948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.450955 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.450996 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.474441 (+ 23445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.475156 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.475160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.475162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.475280 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.475626 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.475629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.475631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.475944 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.476055 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.476389 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.476544 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:06:42.476799 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.444198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:46311 (local address 127.0.0.1:43308)
0810 18:06:42.444419 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.444435 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.444522 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.444919 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.444922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.445145 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.445267 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.445274 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.446531 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.446535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.447485 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.447498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.448291 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.448295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.448415 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.449060 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.449077 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.450753 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.474666 (+ 23913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.474671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.474674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.475018 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.475403 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.475406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.475408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.475516 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.476055 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.476059 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.476183 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.476304 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.476505 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260810 18:06:42.478435 29159 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.182.124:46311: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543"
member_type: NON_VOTER
last_known_addr {
host: "127.21.182.124"
port: 46311
}
attrs {
promote: true
}
}
dest_uuid: "f3ff08bd6f294102b42e2970087bcad4"
cas_config_opid_index: 5
I20260810 18:06:42.479192 29159 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: NON_VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: true } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } }
I20260810 18:06:42.480567 29783 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ea36ca408c3f4a00b71c76b044ca6543 (127.21.182.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: NON_VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: true } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } } }
I20260810 18:06:42.481914 29931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:42.482141 29783 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.182.124:46311
I20260810 18:06:42.482333 29783 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: NON_VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: true } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } } }
I20260810 18:06:42.482440 29783 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:42.483579 29931 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 1 FOLLOWER]: Refusing update from remote peer f3ff08bd6f294102b42e2970087bcad4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 18:06:42.483588 29790 master.cc:537] Master@127.21.182.124:46311 shutting down...
I20260810 18:06:42.484372 29782 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:06:42.487346 29143 consensus_peers.cc:487] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 -> Peer ea36ca408c3f4a00b71c76b044ca6543 (127.21.182.124:46311): Couldn't send request to peer ea36ca408c3f4a00b71c76b044ca6543. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:06:42.490430 29790 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:42.490899 29790 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:42.491112 29790 tablet_replica.cc:330] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: stopping tablet replica
I20260810 18:06:42.494163 29522 heartbeater.cc:503] Master 127.21.182.126:37161 was elected leader, sending a full tablet report...
I20260810 18:06:42.495424 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.480944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:35871 (local address 127.0.0.1:38426)
0810 18:06:42.489435 (+ 8491us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.489455 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.489581 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.490535 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.490539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.490563 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:42.490696 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.490706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.492082 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.492086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.493535 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.493550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.494904 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.494909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.495016 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.495083 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.495171 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.495265 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8385}
I20260810 18:06:42.495862 29967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.481220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38426 (local address 127.21.182.125:35871)
0810 18:06:42.488787 (+ 7567us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.488792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.489698 (+ 906us) server_negotiation.cc:407] Connection header received
0810 18:06:42.489774 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.489778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.489867 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.490346 (+ 479us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:42.490857 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.491924 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.493727 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.494476 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.494617 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.494820 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.495611 (+ 791us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.495724 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7433}
I20260810 18:06:42.496789 29833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 LEARNER]: Refusing update from remote peer f3ff08bd6f294102b42e2970087bcad4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 18:06:42.497344 29782 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: NON_VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 18:06:42.499771 29833 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ea36ca408c3f4a00b71c76b044ca6543 (127.21.182.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: NON_VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: true } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } } }
I20260810 18:06:42.502071 29993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: NON_VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: true } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } } }
I20260810 18:06:42.502334 29993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: This master's current role is: LEARNER
I20260810 18:06:42.502532 29875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3ff08bd6f294102b42e2970087bcad4. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: NON_VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: true } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } } }
I20260810 18:06:42.502662 29875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: This master's current role is: LEARNER
I20260810 18:06:42.503556 29782 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4: attempting to promote NON_VOTER 31a294944c674d20ac6295f3ebbd2f9d to VOTER
I20260810 18:06:42.503858 29790 master.cc:552] Master@127.21.182.124:46311 shutdown complete.
I20260810 18:06:42.503938 29790 master_runner.cc:305] Clearing existing system tablet
I20260810 18:06:42.504204 29790 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:42.504254 29790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:42.504171 29782 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } }
I20260810 18:06:42.506058 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.505441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:46311 (local address 127.0.0.1:43320)
0810 18:06:42.505873 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.505964 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260810 18:06:42.506639 29833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 LEARNER]: Refusing update from remote peer f3ff08bd6f294102b42e2970087bcad4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 18:06:42.507377 29786 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 18:06:42.508273 29790 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:06:42.509027 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.507882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:46311 (local address 127.0.0.1:43328)
0810 18:06:42.508878 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.508942 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":905}
I20260810 18:06:42.509235 29995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:42.509446 29790 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:42.509552 29790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:06:42.509609 29790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/wal
uuid: "ea36ca408c3f4a00b71c76b044ca6543"
format_stamp: "Formatted at 2026-08-10 18:06:42 on dist-test-slave-27sr"
server_key: "d2334155366a3ea71b8c80e2c18e64a8"
server_key_iv: "69748db13fe22d6d55d2a5a246fdc7a9"
server_key_version: "encryptionkey@0"
W20260810 18:06:42.510795 29143 consensus_peers.cc:487] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 -> Peer ea36ca408c3f4a00b71c76b044ca6543 (127.21.182.124:46311): Couldn't send request to peer ea36ca408c3f4a00b71c76b044ca6543. Status: Network error: Client connection negotiation failed: client connection to 127.21.182.124:46311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:06:42.511292 29786 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 31a294944c674d20ac6295f3ebbd2f9d (127.21.182.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } } }
I20260810 18:06:42.511798 29790 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:42.513006 29786 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } } }
I20260810 18:06:42.513120 29786 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:42.513259 29786 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } } }
I20260810 18:06:42.513331 29786 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:42.514302 29833 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 31a294944c674d20ac6295f3ebbd2f9d (127.21.182.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } } }
I20260810 18:06:42.514513 29790 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 18:06:42.514611 29790 log.cc:1196] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 18:06:42.514865 29790 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: Deleting consensus metadata
I20260810 18:06:42.515043 29790 master_runner.cc:315] Copying system tablet from 127.21.182.126:37161
I20260810 18:06:42.516036 29875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } } }
I20260810 18:06:42.516160 29875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:06:42.516695 29790 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: tablet copy: Beginning tablet copy session from remote peer at address 127.21.182.126:37161
I20260810 18:06:42.527740 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.517473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:33964)
0810 18:06:42.517984 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.518000 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.518119 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.518454 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.518457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.518708 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.518863 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.518870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.520146 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.520151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.521079 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.521091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.521716 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.521719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.521821 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.522966 (+ 1145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.522982 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.523489 (+ 507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.525917 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.525921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.525924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.526207 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.526546 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.526550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.526552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.526652 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.527125 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.527128 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.527242 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.527357 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.527529 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":68,"threads_started":1}
I20260810 18:06:42.528208 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.517635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.21.182.126:37161)
0810 18:06:42.517799 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.517805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.518093 (+ 288us) server_negotiation.cc:407] Connection header received
0810 18:06:42.518214 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.518221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.518291 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.518391 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.519002 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.520024 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.521219 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.521584 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.521713 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.523647 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.523652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.523655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.523694 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.525774 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.526323 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.526327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.526330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.526438 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.526758 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.526762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.526764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.527017 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.527127 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.527440 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.527587 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 18:06:42.528870 29179 tablet_copy_service.cc:140] P f3ff08bd6f294102b42e2970087bcad4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ea36ca408c3f4a00b71c76b044ca6543 ({username='oryx', principal='oryx/127.21.182.124@KRBTEST.COM'} at 127.0.0.1:33964)
I20260810 18:06:42.528983 29179 tablet_copy_service.cc:161] P f3ff08bd6f294102b42e2970087bcad4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ea36ca408c3f4a00b71c76b044ca6543 at {username='oryx', principal='oryx/127.21.182.124@KRBTEST.COM'} at 127.0.0.1:33964: session id = ea36ca408c3f4a00b71c76b044ca6543-00000000000000000000000000000000
I20260810 18:06:42.530247 29179 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:06:42.531823 29790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 18:06:42.534713 29790 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: tablet copy: Starting download of 0 data blocks...
I20260810 18:06:42.534946 29790 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: tablet copy: Starting download of 1 WAL segments...
I20260810 18:06:42.536834 29790 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:06:42.539407 29179 tablet_copy_service.cc:342] P f3ff08bd6f294102b42e2970087bcad4: Request end of tablet copy session ea36ca408c3f4a00b71c76b044ca6543-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.182.124@KRBTEST.COM'} at 127.0.0.1:33964
I20260810 18:06:42.539521 29179 tablet_copy_service.cc:433] P f3ff08bd6f294102b42e2970087bcad4: ending tablet copy session ea36ca408c3f4a00b71c76b044ca6543-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ea36ca408c3f4a00b71c76b044ca6543
I20260810 18:06:42.549269 29790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:42.551160 30004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:42.552888 30005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:42.553237 30007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:42.557041 29790 server_base.cc:1034] running on GCE node
I20260810 18:06:42.557301 29790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:42.557708 29790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:42.558867 29790 hybrid_clock.cc:648] HybridClock initialized: now 1786385202558829 us; error 61 us; skew 500 ppm
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385202, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:42.561363 29790 init.cc:377] Logged in from keytab as oryx/127.21.182.124@KRBTEST.COM (short username oryx)
I20260810 18:06:42.562083 29790 webserver.cc:460] Webserver started at http://127.21.182.124:34807/ using document root <none> and password file <none>
I20260810 18:06:42.562378 29790 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:42.562439 29790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:42.564304 29790 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 18:06:42.565631 30015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:42.565938 29790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:06:42.566082 29790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/wal
uuid: "ea36ca408c3f4a00b71c76b044ca6543"
format_stamp: "Formatted at 2026-08-10 18:06:42 on dist-test-slave-27sr"
server_key: "d2334155366a3ea71b8c80e2c18e64a8"
server_key_iv: "69748db13fe22d6d55d2a5a246fdc7a9"
server_key_version: "encryptionkey@0"
I20260810 18:06:42.566298 29790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:42.597677 29790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:42.600425 29790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:42.606310 29790 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.124:46311
I20260810 18:06:42.606940 30070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.124:46311 every 8 connection(s)
I20260810 18:06:42.607249 29790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/master-2/data/info.pb
I20260810 18:06:42.608913 30072 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:06:42.609928 30072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: Bootstrap starting.
W20260810 18:06:42.616837 29991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:42.617120 29991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:42.617214 29991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 18:06:42.620514 30072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:06:42.620877 30072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: Bootstrap complete.
I20260810 18:06:42.621532 30072 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } }
I20260810 18:06:42.621692 30072 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea36ca408c3f4a00b71c76b044ca6543, State: Initialized, Role: LEARNER
I20260810 18:06:42.621846 30072 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } }
I20260810 18:06:42.622251 30079 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } } }
I20260810 18:06:42.622378 30079 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [sys.catalog]: This master's current role is: LEARNER
I20260810 18:06:42.622444 30072 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 18:06:42.622803 29991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:42.622902 29991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:42.623045 29991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:42.625918 30090 catalog_manager.cc:1259] Loaded cluster ID: ca89d301d2e14245aea25dc5cf52a4a8
I20260810 18:06:42.625993 30090 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: loading cluster ID for follower catalog manager: success
I20260810 18:06:42.627681 30090 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: acquiring CA information for follower catalog manager: success
I20260810 18:06:42.628911 30090 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:06:42.630846 29991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.65.keytab
--principal=oryx/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:42177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=42377
--tserver_master_addrs=127.21.182.126:37161,127.21.182.125:35871,127.21.182.124:46311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:42.632200 29991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:42.633414 29991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:42.644688 30095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:42.644742 30092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:42.644994 29991 server_base.cc:1034] running on GCE node
W20260810 18:06:42.644713 30093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:42.645630 29991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:42.646377 29991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:42.647578 29991 hybrid_clock.cc:648] HybridClock initialized: now 1786385202647557 us; error 36 us; skew 500 ppm
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385202, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:42.650866 29991 init.cc:377] Logged in from keytab as oryx/127.21.182.65@KRBTEST.COM (short username oryx)
I20260810 18:06:42.651859 29991 webserver.cc:460] Webserver started at http://127.21.182.65:42377/ using document root <none> and password file <none>
I20260810 18:06:42.652558 29991 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:42.652680 29991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:42.656963 29991 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:06:42.659282 30103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:42.660671 29991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:42.660820 29991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "b9f6e36bb9254c95baa2cc33af78a1fe"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "e22e4e9d317aef326fc9a086fda8db96"
server_key_iv: "1e07e4c58c8d6b886d89e93acc4ed1b0"
server_key_version: "encryptionkey@0"
I20260810 18:06:42.661406 29991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:42.686170 29991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:42.690258 29991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:42.690495 29991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:42.691197 29991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:42.692288 29991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:42.692368 29991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:42.692445 29991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:42.692488 29991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:42.704433 29991 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:42177
I20260810 18:06:42.704491 30231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:42177 every 8 connection(s)
I20260810 18:06:42.705780 29991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:42.711038 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 29991
I20260810 18:06:42.711261 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 29378
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385202, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385202, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.124@KRBTEST.COM
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385202, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.125@KRBTEST.COM
I20260810 18:06:42.726543 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.708065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:38143 (local address 127.21.182.126:37161)
0810 18:06:42.708211 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.708216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.709522 (+ 1306us) server_negotiation.cc:407] Connection header received
0810 18:06:42.710627 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.710631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.710700 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.710789 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.713213 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.713987 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.715933 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.716270 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.716374 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.720794 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.720801 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.720808 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.720847 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.723246 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.724331 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.724335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.724337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.724474 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.724890 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.724893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.724895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.725143 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.725261 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.726171 (+ 910us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.726338 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:06:42.727978 30236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.708016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:53779 (local address 127.21.182.124:46311)
0810 18:06:42.708307 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.708312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.710696 (+ 2384us) server_negotiation.cc:407] Connection header received
0810 18:06:42.710756 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.710761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.710822 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.710906 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.714892 (+ 3986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.715946 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.717022 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.717434 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.717551 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.720932 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.720935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.720938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.720969 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.723727 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.725304 (+ 1577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.725311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.725316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.725460 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.725910 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.725916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.725921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.726254 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.726378 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.727444 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.727647 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
I20260810 18:06:42.729451 30238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.708532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.65:38143)
0810 18:06:42.709414 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.709429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.710408 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.711147 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.711154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.712932 (+ 1778us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.713064 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.713071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.714129 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.714132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.715684 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.715696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.716567 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.716571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.716799 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.717649 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.717674 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.720521 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.723472 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.723482 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.723508 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.724181 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.724564 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.724570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.724574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.724766 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.725330 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.725338 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.725730 (+ 392us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.727010 (+ 1280us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.728488 (+ 1478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":1233,"thread_start_us":182,"threads_started":1}
I20260810 18:06:42.729830 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.708532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:46311 (local address 127.21.182.65:53779)
0810 18:06:42.709127 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.709184 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.710394 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.711144 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.711154 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.714600 (+ 3446us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.714721 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.714728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.716070 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.716073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.716875 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.716885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.717692 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.717695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.717794 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.719050 (+ 1256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.719064 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.720769 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.723863 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.723870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.723875 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.725138 (+ 1263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.725611 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.725617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.725622 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.725754 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.726440 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.726447 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.726587 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.727125 (+ 538us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.727924 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":4,"thread_start_us":625,"threads_started":1}
I20260810 18:06:42.730388 29967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.710020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:34095 (local address 127.21.182.125:35871)
0810 18:06:42.710197 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.710202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.710338 (+ 136us) server_negotiation.cc:407] Connection header received
0810 18:06:42.710632 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.710635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.710727 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.710795 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.713003 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.713852 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.721339 (+ 7487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.721764 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.721869 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.726325 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.726340 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.726343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.726376 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.728316 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.728839 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.728843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.728845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.728945 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.729309 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.729312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.729314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.729588 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.729688 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.730054 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.730224 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 18:06:42.731101 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.709746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:35871 (local address 127.21.182.65:34095)
0810 18:06:42.710243 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.710255 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.710394 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.711144 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.711154 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.711782 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.712778 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.712805 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.714046 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.714052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.718847 (+ 4795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.718862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.722523 (+ 3661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.722530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.722685 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.723931 (+ 1246us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.723960 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.726134 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.728461 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.728464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.728467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.728709 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.729058 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.729061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.729063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.729146 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.729749 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.729752 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.729850 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.730832 (+ 982us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.730976 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":62,"thread_start_us":147,"threads_started":1}
I20260810 18:06:42.732439 30234 heartbeater.cc:347] Connected to a master server at 127.21.182.125:35871
I20260810 18:06:42.732746 30234 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:42.733444 30234 heartbeater.cc:511] Master 127.21.182.125:35871 requested a full tablet report, sending...
I20260810 18:06:42.734213 30233 heartbeater.cc:347] Connected to a master server at 127.21.182.126:37161
I20260810 18:06:42.734323 30233 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:42.734380 30232 heartbeater.cc:347] Connected to a master server at 127.21.182.124:46311
I20260810 18:06:42.734450 30232 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:42.734553 30233 heartbeater.cc:511] Master 127.21.182.126:37161 requested a full tablet report, sending...
I20260810 18:06:42.734571 30232 heartbeater.cc:511] Master 127.21.182.124:46311 requested a full tablet report, sending...
I20260810 18:06:42.735432 29822 ts_manager.cc:194] Registered new tserver with Master: b9f6e36bb9254c95baa2cc33af78a1fe (127.21.182.65:42177)
I20260810 18:06:42.735471 29159 ts_manager.cc:194] Re-registered known tserver with Master: b9f6e36bb9254c95baa2cc33af78a1fe (127.21.182.65:42177)
I20260810 18:06:42.735795 30033 ts_manager.cc:194] Registered new tserver with Master: b9f6e36bb9254c95baa2cc33af78a1fe (127.21.182.65:42177)
I20260810 18:06:42.736326 29159 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:38143
I20260810 18:06:42.738457 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:36975
--local_ip_for_outbound_sockets=127.21.182.66
--tserver_master_addrs=127.21.182.126:37161
--webserver_port=42321
--webserver_interface=127.21.182.66
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.66.keytab
--principal=oryx/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.182.126:37161,127.21.182.125:35871
--tserver_master_addrs=127.21.182.126:37161,127.21.182.125:35871,127.21.182.124:46311 with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:06:42.771703 29673 heartbeater.cc:503] Master 127.21.182.126:37161 was elected leader, sending a full tablet report...
W20260810 18:06:42.866832 30240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:42.867146 30240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:42.867223 30240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:42.871538 30240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:42.871619 30240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:42.871722 30240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:42.877616 30240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.66.keytab
--principal=oryx/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:36975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=42321
--tserver_master_addrs=127.21.182.126:37161,127.21.182.125:35871,127.21.182.124:46311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:42.878931 30240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:42.880199 30240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:42.891970 30248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:42.891970 30246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:42.892198 30240 server_base.cc:1034] running on GCE node
W20260810 18:06:42.892360 30245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:42.892863 30240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:42.893668 30240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:42.894889 30240 hybrid_clock.cc:648] HybridClock initialized: now 1786385202894810 us; error 98 us; skew 500 ppm
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385202, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:42.898232 30240 init.cc:377] Logged in from keytab as oryx/127.21.182.66@KRBTEST.COM (short username oryx)
I20260810 18:06:42.899401 30240 webserver.cc:460] Webserver started at http://127.21.182.66:42321/ using document root <none> and password file <none>
I20260810 18:06:42.900136 30240 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:42.900266 30240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:42.904801 30240 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:06:42.907330 30256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:42.908805 30240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:42.908967 30240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "aba4923d0f9b498289e62debae97813e"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "76035a124a1180bd0f95b04a32c805d5"
server_key_iv: "cb4137fe0e794fb2c095f287c38c02ba"
server_key_version: "encryptionkey@0"
I20260810 18:06:42.909626 30240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:42.919751 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.915872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:46311 (local address 127.0.0.1:43342)
0810 18:06:42.916098 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.916118 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.916247 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.916602 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.916605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.916625 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:42.916791 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.916797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.917706 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.917713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.918503 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.918513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.919304 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.919308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.919403 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.919428 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.919501 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.919594 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260810 18:06:42.919766 30236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.916051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.21.182.124:46311)
0810 18:06:42.916221 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.916227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.916247 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:06:42.916370 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.916374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.916442 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.916555 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:42.917013 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.917573 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.918664 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.919149 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.919304 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.919471 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.919513 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.919592 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:06:42.920739 30044 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 18:06:42.921249 30079 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3ff08bd6f294102b42e2970087bcad4. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } } }
I20260810 18:06:42.921497 30079 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [sys.catalog]: This master's current role is: LEARNER
I20260810 18:06:42.942477 30240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:42.946432 30240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:42.946623 30240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:42.947367 30240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:42.948475 30240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:42.948532 30240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:42.948580 30240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:42.948596 30240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:42.960223 30240 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:36975
I20260810 18:06:42.960336 30385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:36975 every 8 connection(s)
I20260810 18:06:42.961486 30240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:06:42.967407 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 30240
I20260810 18:06:42.967847 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 29527
I20260810 18:06:42.984059 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:40943
--local_ip_for_outbound_sockets=127.21.182.67
--tserver_master_addrs=127.21.182.126:37161
--webserver_port=42605
--webserver_interface=127.21.182.67
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.67.keytab
--principal=oryx/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.182.126:37161,127.21.182.125:35871
--tserver_master_addrs=127.21.182.126:37161,127.21.182.125:35871,127.21.182.124:46311 with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385202, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.125@KRBTEST.COM
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385202, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
Aug 10 18:06:42 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385202, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.124@KRBTEST.COM
I20260810 18:06:42.991539 29967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.964353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:40175 (local address 127.21.182.125:35871)
0810 18:06:42.964515 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.964519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.965588 (+ 1069us) server_negotiation.cc:407] Connection header received
0810 18:06:42.967233 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.967237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.967301 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.967388 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.977690 (+ 10302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.978694 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.981277 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.981745 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.981860 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.985999 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.986007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.986010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.986054 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.988217 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.989013 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.989021 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.989026 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.989137 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.989649 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.989656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.989662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.989983 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.990104 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.991069 (+ 965us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.991261 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:06:42.992400 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.963738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:46737 (local address 127.21.182.126:37161)
0810 18:06:42.963949 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.963955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.965294 (+ 1339us) server_negotiation.cc:407] Connection header received
0810 18:06:42.966887 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.966891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.966962 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.967080 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.977702 (+ 10622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.978661 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.981261 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.981667 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.981764 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.986559 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.986564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.986568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.986602 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.989860 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.990454 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.990458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.990460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.990562 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.990983 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.990987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.990989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.991246 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.991351 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.991740 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.991878 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:06:42.992998 30236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.965880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:49957 (local address 127.21.182.124:46311)
0810 18:06:42.966035 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:42.966041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:42.966347 (+ 306us) server_negotiation.cc:407] Connection header received
0810 18:06:42.966907 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:42.966911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:42.966962 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:42.967055 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:42.977687 (+ 10632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.978661 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.981265 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.981664 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.981764 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.988438 (+ 6674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:42.988445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:42.988452 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:42.988498 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:42.990535 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.991120 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.991124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.991126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.991232 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:42.991589 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:42.991596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:42.991601 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:42.991848 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:42.992125 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:42.992677 (+ 552us) server_negotiation.cc:299] Negotiation successful
0810 18:06:42.992828 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:06:42.993971 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.965774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:46311 (local address 127.21.182.66:49957)
0810 18:06:42.966259 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.966274 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.966690 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.967301 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.967314 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.976148 (+ 8834us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.977388 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.977423 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.978887 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.978891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.981057 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.981074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.982031 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.982036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.982477 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.986215 (+ 3738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.986238 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.988205 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.990675 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.990678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.990680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.990991 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.991367 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.991371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.991373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.991467 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.992199 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.992202 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.992338 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.993271 (+ 933us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.993428 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":782,"spinlock_wait_cycles":28672,"thread_start_us":98,"threads_started":1}
I20260810 18:06:42.993975 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.964323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:35871 (local address 127.21.182.66:40175)
0810 18:06:42.965478 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.965493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.966623 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.967531 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.967537 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.976318 (+ 8781us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.977387 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.977424 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.978889 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.978896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.981081 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.981098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.982075 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.982081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.982472 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.983480 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.983501 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.985685 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.988388 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.988399 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.988425 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.988867 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.989285 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.989291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.989297 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.989519 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.990197 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.990221 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.990641 (+ 420us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.991848 (+ 1207us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.992456 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"mutex_wait_us":547,"spinlock_wait_cycles":96512,"thread_start_us":219,"threads_started":1}
I20260810 18:06:42.994597 30390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:42.964324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.66:46737)
0810 18:06:42.965122 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:42.965172 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:42.966625 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:42.967304 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:42.967315 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:42.976148 (+ 8833us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:42.977389 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.977423 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.978887 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.978891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.981083 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:42.981090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:42.982031 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:42.982036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:42.982475 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:42.983831 (+ 1356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:42.983862 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:42.986349 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:42.990027 (+ 3678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.990030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.990032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.990309 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.990681 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:42.990700 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:42.990702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:42.990846 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:42.991406 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:42.991412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:42.991517 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 18:06:42.992804 (+ 1287us) client_negotiation.cc:240] Negotiation successful
0810 18:06:42.992960 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":761,"spinlock_wait_cycles":56064,"thread_start_us":873,"threads_started":1}
I20260810 18:06:42.996114 30387 heartbeater.cc:347] Connected to a master server at 127.21.182.126:37161
I20260810 18:06:42.996115 30388 heartbeater.cc:347] Connected to a master server at 127.21.182.125:35871
I20260810 18:06:42.996556 30388 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:42.996556 30387 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:42.996115 30386 heartbeater.cc:347] Connected to a master server at 127.21.182.124:46311
I20260810 18:06:42.997331 30387 heartbeater.cc:511] Master 127.21.182.126:37161 requested a full tablet report, sending...
I20260810 18:06:42.997422 30386 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:42.997334 30388 heartbeater.cc:511] Master 127.21.182.125:35871 requested a full tablet report, sending...
I20260810 18:06:42.997632 30386 heartbeater.cc:511] Master 127.21.182.124:46311 requested a full tablet report, sending...
I20260810 18:06:42.998821 29822 ts_manager.cc:194] Registered new tserver with Master: aba4923d0f9b498289e62debae97813e (127.21.182.66:36975)
I20260810 18:06:42.998836 29159 ts_manager.cc:194] Re-registered known tserver with Master: aba4923d0f9b498289e62debae97813e (127.21.182.66:36975)
I20260810 18:06:42.999349 30033 ts_manager.cc:194] Registered new tserver with Master: aba4923d0f9b498289e62debae97813e (127.21.182.66:36975)
I20260810 18:06:42.999650 29159 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:46737
W20260810 18:06:43.120290 30393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:43.120641 30393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:43.120690 30393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:43.125260 30393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:43.125345 30393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:43.125411 30393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:43.131096 30393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.67.keytab
--principal=oryx/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:40943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=42605
--tserver_master_addrs=127.21.182.126:37161,127.21.182.125:35871,127.21.182.124:46311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:43.132475 30393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:43.133553 30393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:43.144330 30398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:43.144330 30399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:43.144469 30393 server_base.cc:1034] running on GCE node
W20260810 18:06:43.144342 30401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:43.145092 30393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:43.145855 30393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:43.147058 30393 hybrid_clock.cc:648] HybridClock initialized: now 1786385203147017 us; error 53 us; skew 500 ppm
Aug 10 18:06:43 dist-test-slave-27sr krb5kdc[29110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385203, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:43.150451 30393 init.cc:377] Logged in from keytab as oryx/127.21.182.67@KRBTEST.COM (short username oryx)
I20260810 18:06:43.151404 30393 webserver.cc:460] Webserver started at http://127.21.182.67:42605/ using document root <none> and password file <none>
I20260810 18:06:43.152097 30393 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:43.152194 30393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:43.156633 30393 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:06:43.159226 30408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:43.160765 30393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:43.160920 30393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "28ec792c34e74d42a3c29be691aac1ca"
format_stamp: "Formatted at 2026-08-10 18:06:41 on dist-test-slave-27sr"
server_key: "83bd121de3600f4d900f5e38b47c4bac"
server_key_iv: "d8fabd780212820eff0a03e888a103f2"
server_key_version: "encryptionkey@0"
I20260810 18:06:43.161634 30393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:43.180598 30393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:43.184043 30393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:43.184264 30393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:43.185014 30393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:43.185982 30393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:43.186034 30393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:43.186107 30393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:43.186137 30393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:43.196861 30393 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:40943
I20260810 18:06:43.196870 30534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:40943 every 8 connection(s)
I20260810 18:06:43.198303 30393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:06:43.204515 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 30393
Aug 10 18:06:43 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385203, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.124@KRBTEST.COM
Aug 10 18:06:43 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385203, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
Aug 10 18:06:43 dist-test-slave-27sr krb5kdc[29110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385203, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.125@KRBTEST.COM
I20260810 18:06:43.220378 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.200489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:43559 (local address 127.21.182.126:37161)
0810 18:06:43.200677 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.200684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.201741 (+ 1057us) server_negotiation.cc:407] Connection header received
0810 18:06:43.203204 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.203209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.203269 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.203366 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:43.205663 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.206444 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.208523 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.208868 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.208954 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.213513 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:43.213517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:43.213520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:43.213547 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:43.217755 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.218934 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.218938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.218940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.219038 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.219337 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.219340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.219342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.219553 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:43.219645 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.220089 (+ 444us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.220218 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 18:06:43.220984 30236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.200959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:37825 (local address 127.21.182.124:46311)
0810 18:06:43.201096 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.201101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.202726 (+ 1625us) server_negotiation.cc:407] Connection header received
0810 18:06:43.203348 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.203356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.203423 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.203505 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:43.205460 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.206326 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.208540 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.208873 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.209012 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.212816 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:43.212822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:43.212828 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:43.212864 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:43.215693 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.217853 (+ 2160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.217858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.217861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.217980 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.218499 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.218504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.218507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.218788 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:43.218888 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.220677 (+ 1789us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.220812 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:06:43.221339 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.200987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:46311 (local address 127.21.182.67:37825)
0810 18:06:43.202602 (+ 1615us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:43.202616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:43.202947 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:43.203619 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:43.203622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:43.204289 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:43.205266 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.205289 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.206524 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.206530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.208371 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.208392 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.209066 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.209070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.209261 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.210176 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:43.210204 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:43.212586 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:43.215832 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:43.215843 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:43.215868 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:43.216307 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:43.218111 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:43.218116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:43.218121 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:43.218392 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:43.218950 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:43.218959 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:43.219378 (+ 419us) client_negotiation.cc:769] Sending connection context
0810 18:06:43.219629 (+ 251us) client_negotiation.cc:240] Negotiation successful
0810 18:06:43.220103 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1522,"mutex_wait_us":212,"spinlock_wait_cycles":17024,"thread_start_us":229,"threads_started":1}
I20260810 18:06:43.221666 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.200987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.67:43559)
0810 18:06:43.201556 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:43.201611 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:43.202947 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:43.203582 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:43.203593 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:43.204289 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:43.205525 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.205531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.206588 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.206591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.208367 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.208377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.209126 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.209129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.209265 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.210179 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:43.210201 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:43.213348 (+ 3147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:43.218524 (+ 5176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:43.218526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:43.218528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:43.218790 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:43.219142 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:43.219144 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:43.219146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:43.219232 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:43.219690 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:43.219692 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:43.219780 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 18:06:43.219918 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 18:06:43.221211 (+ 1293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":1469,"spinlock_wait_cycles":21120,"thread_start_us":183,"threads_started":1}
I20260810 18:06:43.222824 30540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.202185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:35871 (local address 127.21.182.67:42923)
0810 18:06:43.204892 (+ 2707us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:43.204908 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:43.205075 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:43.206991 (+ 1916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:43.206994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:43.207231 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:43.207501 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.207513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.208552 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.208555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.209374 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.209382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.210060 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.210064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.210188 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.210617 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:43.210631 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:43.214247 (+ 3616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:43.219584 (+ 5337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:43.219589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:43.219592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:43.219860 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:43.220839 (+ 979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:43.220844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:43.220847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:43.220965 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:43.222263 (+ 1298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:43.222266 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:43.222399 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 18:06:43.222495 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 18:06:43.222646 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2630,"mutex_wait_us":78,"spinlock_wait_cycles":896,"thread_start_us":247,"threads_started":1}
I20260810 18:06:43.223547 29967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.202315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:42923 (local address 127.21.182.125:35871)
0810 18:06:43.202437 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.202442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.205005 (+ 2563us) server_negotiation.cc:407] Connection header received
0810 18:06:43.205196 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.205199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.205250 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.205319 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:43.207657 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.208425 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.209504 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.209803 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.209909 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.214431 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:43.214435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:43.214438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:43.214467 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:43.219439 (+ 4972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.220034 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.220040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.220043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.220141 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.221307 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.221311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.221313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.221568 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:43.221653 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.223212 (+ 1559us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.223366 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:06:43.222949 30535 heartbeater.cc:347] Connected to a master server at 127.21.182.124:46311
I20260810 18:06:43.223941 30535 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:43.222962 30536 heartbeater.cc:347] Connected to a master server at 127.21.182.126:37161
I20260810 18:06:43.224555 30536 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:43.224718 30535 heartbeater.cc:511] Master 127.21.182.124:46311 requested a full tablet report, sending...
I20260810 18:06:43.224779 30536 heartbeater.cc:511] Master 127.21.182.126:37161 requested a full tablet report, sending...
I20260810 18:06:43.226070 29159 ts_manager.cc:194] Re-registered known tserver with Master: 28ec792c34e74d42a3c29be691aac1ca (127.21.182.67:40943)
I20260810 18:06:43.226142 30537 heartbeater.cc:347] Connected to a master server at 127.21.182.125:35871
I20260810 18:06:43.226168 30033 ts_manager.cc:194] Registered new tserver with Master: 28ec792c34e74d42a3c29be691aac1ca (127.21.182.67:40943)
I20260810 18:06:43.226241 30537 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:43.226431 30537 heartbeater.cc:511] Master 127.21.182.125:35871 requested a full tablet report, sending...
I20260810 18:06:43.226910 29159 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:43559
I20260810 18:06:43.227157 29822 ts_manager.cc:194] Registered new tserver with Master: 28ec792c34e74d42a3c29be691aac1ca (127.21.182.67:40943)
I20260810 18:06:43.228454 30547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.209931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33968 (local address 127.21.182.126:37161)
0810 18:06:43.213372 (+ 3441us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.213377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.213409 (+ 32us) server_negotiation.cc:407] Connection header received
0810 18:06:43.213456 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.213461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.213515 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.213604 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:43.214760 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.215790 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.217131 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.217491 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.217580 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.219823 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:43.219839 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:43.219841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:43.219868 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:43.226191 (+ 6323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.226713 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.226717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.226718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.226814 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.227141 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.227144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.227145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.227365 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:43.227497 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.228152 (+ 655us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.228289 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3352,"thread_start_us":82,"threads_started":1}
I20260810 18:06:43.228600 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.211300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.21.182.124:46311)
0810 18:06:43.211564 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.211569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.212851 (+ 1282us) server_negotiation.cc:407] Connection header received
0810 18:06:43.213072 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.213077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.213139 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.213210 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:43.214412 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.215552 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.217087 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.217476 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.217571 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.221475 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:43.221479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:43.221482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:43.221506 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:43.224696 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.225234 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.225238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.225239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.225349 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.226930 (+ 1581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.226933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.226935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.227169 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:43.227298 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.228261 (+ 963us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.228414 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
I20260810 18:06:43.230945 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.210571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38438 (local address 127.21.182.125:35871)
0810 18:06:43.212202 (+ 1631us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.212205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.212216 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:06:43.212245 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.212247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.212333 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.212429 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:43.215189 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.216003 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.217130 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.217474 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.217577 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.223023 (+ 5446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:43.223028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:43.223031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:43.223057 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:43.229002 (+ 5945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.229486 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.229491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.229493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.229593 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.229934 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.229938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.229939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.230215 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:43.230380 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.230649 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.230790 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1554,"thread_start_us":80,"threads_started":1}
I20260810 18:06:43.231972 29159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:06:43.235134 29159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:43.238163 29786 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: NON_VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: true } } }
I20260810 18:06:43.238334 29786 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:43.242249 30556 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4: attempting to promote NON_VOTER ea36ca408c3f4a00b71c76b044ca6543 to VOTER
I20260810 18:06:43.242668 30556 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } }
I20260810 18:06:43.244068 30044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 1 LEARNER]: Refusing update from remote peer f3ff08bd6f294102b42e2970087bcad4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 18:06:43.244091 29833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 FOLLOWER]: Refusing update from remote peer f3ff08bd6f294102b42e2970087bcad4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 18:06:43.244505 29786 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 18:06:43.244807 29786 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 18:06:43.246747 30554 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ea36ca408c3f4a00b71c76b044ca6543 (127.21.182.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } } }
I20260810 18:06:43.247367 29833 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ea36ca408c3f4a00b71c76b044ca6543 (127.21.182.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } } }
I20260810 18:06:43.247593 30044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ea36ca408c3f4a00b71c76b044ca6543 (127.21.182.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } } }
I20260810 18:06:43.248517 30554 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } } }
I20260810 18:06:43.248638 30554 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:43.248687 29786 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } } }
I20260810 18:06:43.248787 29786 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:43.248770 30554 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } } }
I20260810 18:06:43.248893 30554 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:43.249601 30564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } } }
I20260810 18:06:43.249912 30564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:06:43.251441 30079 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3ff08bd6f294102b42e2970087bcad4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } } }
I20260810 18:06:43.251576 30079 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:06:43.258443 29776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.252135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:42177 (local address 127.0.0.1:48398)
0810 18:06:43.252499 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:43.252517 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:43.252605 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:43.253786 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:43.253789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:43.253807 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:43.253941 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.253951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.255565 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.255568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.256830 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.256843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.258030 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.258034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.258147 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.258171 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:06:43.258242 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 18:06:43.258317 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":18}
I20260810 18:06:43.258846 30565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.252795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48398 (local address 127.21.182.65:42177)
0810 18:06:43.253147 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.253155 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.253181 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:06:43.253278 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.253284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.253600 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.253831 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:43.254099 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.255282 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.256985 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.257888 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.258114 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.258260 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.258321 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.258428 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":133,"threads_started":1}
I20260810 18:06:43.259003 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.252803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:36975 (local address 127.0.0.1:58358)
0810 18:06:43.253506 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:43.253521 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:43.253628 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:43.254749 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:43.254752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:43.254771 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:43.254872 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.254877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.256439 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.256444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.257467 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.257479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.258547 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.258552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.258681 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.258716 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 18:06:43.258779 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 18:06:43.258855 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":28,"thread_start_us":127,"threads_started":1}
I20260810 18:06:43.259583 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.253810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.21.182.67:40943)
0810 18:06:43.254187 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.254201 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.254224 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:06:43.254341 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.254346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.254520 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.254681 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:43.254869 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.255726 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.257307 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.257878 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.258024 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.258212 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.259089 (+ 877us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.259208 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":112,"threads_started":1}
I20260810 18:06:43.259820 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.253164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:40943 (local address 127.0.0.1:56358)
0810 18:06:43.254118 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:43.254132 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:43.254224 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:43.254653 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:43.254657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:43.254674 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:43.254772 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.254778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.255854 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.255858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.257186 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.257195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.258025 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.258029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.258796 (+ 767us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.258815 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:06:43.258887 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 18:06:43.258960 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":78,"thread_start_us":186,"threads_started":1}
I20260810 18:06:43.259991 30567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.252998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.21.182.66:36975)
0810 18:06:43.253478 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.253490 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.253619 (+ 129us) server_negotiation.cc:407] Connection header received
0810 18:06:43.254234 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.254241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.254444 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.254601 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:43.255001 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.256320 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.257619 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.258415 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.258528 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.259588 (+ 1060us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.259671 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.259778 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":137,"threads_started":1}
I20260810 18:06:43.262501 30459 tablet_service.cc:1467] Processing CreateTablet for tablet 219b679d06004052bf1e39d7ac6a1468 (DEFAULT_TABLE table=test-table [id=cef9d7b1c8bf494293bd442dbced882d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:43.262622 30158 tablet_service.cc:1467] Processing CreateTablet for tablet 219b679d06004052bf1e39d7ac6a1468 (DEFAULT_TABLE table=test-table [id=cef9d7b1c8bf494293bd442dbced882d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:43.262622 30311 tablet_service.cc:1467] Processing CreateTablet for tablet 219b679d06004052bf1e39d7ac6a1468 (DEFAULT_TABLE table=test-table [id=cef9d7b1c8bf494293bd442dbced882d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:43.263767 30459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219b679d06004052bf1e39d7ac6a1468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:43.263767 30311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219b679d06004052bf1e39d7ac6a1468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:43.263767 30158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219b679d06004052bf1e39d7ac6a1468. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:43.272154 30572 tablet_bootstrap.cc:492] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca: Bootstrap starting.
I20260810 18:06:43.273366 30573 tablet_bootstrap.cc:492] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e: Bootstrap starting.
I20260810 18:06:43.273720 30574 tablet_bootstrap.cc:492] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe: Bootstrap starting.
I20260810 18:06:43.275051 30572 tablet_bootstrap.cc:654] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:43.275727 30573 tablet_bootstrap.cc:654] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:43.275970 30572 log.cc:824] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:43.276188 30574 tablet_bootstrap.cc:654] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:43.276679 30573 log.cc:824] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:43.277005 30574 log.cc:824] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:43.278708 30573 tablet_bootstrap.cc:492] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e: No bootstrap required, opened a new log
I20260810 18:06:43.278707 30572 tablet_bootstrap.cc:492] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca: No bootstrap required, opened a new log
I20260810 18:06:43.278949 30573 ts_tablet_manager.cc:1397] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:06:43.278944 30572 ts_tablet_manager.cc:1397] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:06:43.279253 30574 tablet_bootstrap.cc:492] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe: No bootstrap required, opened a new log
I20260810 18:06:43.279480 30574 ts_tablet_manager.cc:1397] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:06:43.282873 30573 raft_consensus.cc:348] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:43.283141 30573 raft_consensus.cc:374] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:43.283231 30573 raft_consensus.cc:729] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba4923d0f9b498289e62debae97813e, State: Initialized, Role: FOLLOWER
I20260810 18:06:43.283764 30573 consensus_queue.cc:260] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:43.284111 30572 raft_consensus.cc:348] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:43.284413 30572 raft_consensus.cc:374] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:43.284479 30572 raft_consensus.cc:729] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ec792c34e74d42a3c29be691aac1ca, State: Initialized, Role: FOLLOWER
I20260810 18:06:43.284998 30573 ts_tablet_manager.cc:1428] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 18:06:43.284992 30572 consensus_queue.cc:260] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:43.285216 30387 heartbeater.cc:503] Master 127.21.182.126:37161 was elected leader, sending a full tablet report...
I20260810 18:06:43.285439 30574 raft_consensus.cc:348] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:43.285727 30574 raft_consensus.cc:374] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:43.285810 30574 raft_consensus.cc:729] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f6e36bb9254c95baa2cc33af78a1fe, State: Initialized, Role: FOLLOWER
I20260810 18:06:43.286139 30572 ts_tablet_manager.cc:1428] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260810 18:06:43.286339 30574 consensus_queue.cc:260] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:43.287492 30574 ts_tablet_manager.cc:1428] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 18:06:43.289017 30233 heartbeater.cc:503] Master 127.21.182.126:37161 was elected leader, sending a full tablet report...
I20260810 18:06:43.289096 30536 heartbeater.cc:503] Master 127.21.182.126:37161 was elected leader, sending a full tablet report...
I20260810 18:06:43.403328 30580 raft_consensus.cc:482] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:43.403625 30580 raft_consensus.cc:504] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:43.405208 30580 leader_election.cc:288] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aba4923d0f9b498289e62debae97813e (127.21.182.66:36975), 28ec792c34e74d42a3c29be691aac1ca (127.21.182.67:40943)
I20260810 18:06:43.411175 30567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.405646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:53101 (local address 127.21.182.66:36975)
0810 18:06:43.405819 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.405823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.405954 (+ 131us) server_negotiation.cc:407] Connection header received
0810 18:06:43.406189 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.406193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.406264 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.406348 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:43.406782 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.407602 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.409717 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.410405 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.410553 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.410612 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.410970 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.411049 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 18:06:43.411182 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.406137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:60733 (local address 127.21.182.67:40943)
0810 18:06:43.406297 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.406301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.406434 (+ 133us) server_negotiation.cc:407] Connection header received
0810 18:06:43.406526 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.406530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.406588 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.406688 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:43.407112 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.407937 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.409717 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.410419 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.410553 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.410612 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.410940 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.411031 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 18:06:43.411199 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.405479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:36975 (local address 127.21.182.65:53101)
0810 18:06:43.405852 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:43.405869 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:43.406052 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:43.406418 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:43.406425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:43.406455 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:43.406605 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.406615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.407845 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.407848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.409549 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.409562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.410574 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.410578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.410826 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.410859 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 18:06:43.410928 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 18:06:43.410985 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":121,"thread_start_us":170,"threads_started":1}
I20260810 18:06:43.411211 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.405995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:40943 (local address 127.21.182.65:60733)
0810 18:06:43.406331 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:43.406346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:43.406436 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:43.406824 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:43.406830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:43.406852 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:43.406977 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.406982 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.408047 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.408051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.409549 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:43.409562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.410574 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.410578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:43.410741 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.410772 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 18:06:43.410885 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 18:06:43.410976 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":70,"thread_start_us":115,"threads_started":1}
I20260810 18:06:43.412205 30335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "219b679d06004052bf1e39d7ac6a1468" candidate_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aba4923d0f9b498289e62debae97813e" is_pre_election: true
I20260810 18:06:43.412221 30480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "219b679d06004052bf1e39d7ac6a1468" candidate_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ec792c34e74d42a3c29be691aac1ca" is_pre_election: true
I20260810 18:06:43.412581 30335 raft_consensus.cc:2454] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9f6e36bb9254c95baa2cc33af78a1fe in term 0.
I20260810 18:06:43.412581 30480 raft_consensus.cc:2454] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9f6e36bb9254c95baa2cc33af78a1fe in term 0.
I20260810 18:06:43.413129 30108 leader_election.cc:302] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aba4923d0f9b498289e62debae97813e, b9f6e36bb9254c95baa2cc33af78a1fe; no voters:
I20260810 18:06:43.413416 30580 raft_consensus.cc:2781] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:43.413535 30580 raft_consensus.cc:482] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:43.413619 30580 raft_consensus.cc:3037] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:43.415027 30580 raft_consensus.cc:504] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:43.415433 30580 leader_election.cc:288] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [CANDIDATE]: Term 1 election: Requested vote from peers aba4923d0f9b498289e62debae97813e (127.21.182.66:36975), 28ec792c34e74d42a3c29be691aac1ca (127.21.182.67:40943)
I20260810 18:06:43.416018 30335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "219b679d06004052bf1e39d7ac6a1468" candidate_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aba4923d0f9b498289e62debae97813e"
I20260810 18:06:43.416044 30480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "219b679d06004052bf1e39d7ac6a1468" candidate_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ec792c34e74d42a3c29be691aac1ca"
I20260810 18:06:43.416184 30480 raft_consensus.cc:3037] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:43.416168 30335 raft_consensus.cc:3037] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:43.417469 30480 raft_consensus.cc:2454] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9f6e36bb9254c95baa2cc33af78a1fe in term 1.
I20260810 18:06:43.417491 30335 raft_consensus.cc:2454] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9f6e36bb9254c95baa2cc33af78a1fe in term 1.
I20260810 18:06:43.417850 30108 leader_election.cc:302] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28ec792c34e74d42a3c29be691aac1ca, b9f6e36bb9254c95baa2cc33af78a1fe; no voters:
I20260810 18:06:43.418073 30580 raft_consensus.cc:2781] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:43.418375 30580 raft_consensus.cc:686] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [term 1 LEADER]: Becoming Leader. State: Replica: b9f6e36bb9254c95baa2cc33af78a1fe, State: Running, Role: LEADER
I20260810 18:06:43.418913 30580 consensus_queue.cc:237] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:43.423413 29159 catalog_manager.cc:5495] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe reported cstate change: term changed from 0 to 1, leader changed from <none> to b9f6e36bb9254c95baa2cc33af78a1fe (127.21.182.65). New cstate: current_term: 1 leader_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } health_report { overall_health: UNKNOWN } } }
W20260810 18:06:43.451068 30538 tablet.cc:2369] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:06:43.452622 30565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.448533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.21.182.65:42177)
0810 18:06:43.448686 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.448691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.448706 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:06:43.448899 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.448902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.448962 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.449113 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:43.449380 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.449942 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.450940 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.451189 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.451310 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.451689 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:43.452170 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:43.452318 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.452385 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.452457 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 18:06:43.457847 30235 tablet.cc:2369] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:06:43.460096 30335 raft_consensus.cc:1261] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e [term 1 FOLLOWER]: Refusing update from remote peer b9f6e36bb9254c95baa2cc33af78a1fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:43.460114 30480 raft_consensus.cc:1261] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca [term 1 FOLLOWER]: Refusing update from remote peer b9f6e36bb9254c95baa2cc33af78a1fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:43.461046 30583 consensus_queue.cc:1035] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:43.461306 30580 consensus_queue.cc:1035] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:06:43.463315 30389 tablet.cc:2369] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:06:43.471671 30586 mvcc.cc:205] Tried to move back new op lower bound from 7317033793362210816 to 7317033793207275520. Current Snapshot: MvccSnapshot[applied={T|T < 7317033793362210816}]
I20260810 18:06:43.473207 30588 mvcc.cc:205] Tried to move back new op lower bound from 7317033793362210816 to 7317033793207275520. Current Snapshot: MvccSnapshot[applied={T|T < 7317033793362210816}]
I20260810 18:06:43.488298 29158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:43.488538 29158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:43.493744 29158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3ff08bd6f294102b42e2970087bcad4: Sending DeleteTablet for 3 replicas of tablet 219b679d06004052bf1e39d7ac6a1468
I20260810 18:06:43.494706 30158 tablet_service.cc:1514] Processing DeleteTablet for tablet 219b679d06004052bf1e39d7ac6a1468 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:43 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:48398
I20260810 18:06:43.495550 30459 tablet_service.cc:1514] Processing DeleteTablet for tablet 219b679d06004052bf1e39d7ac6a1468 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:43 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:56358
I20260810 18:06:43.495533 30311 tablet_service.cc:1514] Processing DeleteTablet for tablet 219b679d06004052bf1e39d7ac6a1468 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:43 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:58358
I20260810 18:06:43.496053 30597 tablet_replica.cc:330] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca: stopping tablet replica
I20260810 18:06:43.496474 30597 raft_consensus.cc:2229] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:43.496510 30598 tablet_replica.cc:330] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e: stopping tablet replica
I20260810 18:06:43.496851 30598 raft_consensus.cc:2229] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:43.496881 30597 raft_consensus.cc:2258] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:43.497202 30598 raft_consensus.cc:2258] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:43.497611 30596 tablet_replica.cc:330] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe: stopping tablet replica
I20260810 18:06:43.497954 30596 raft_consensus.cc:2229] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:06:43.498536 30596 raft_consensus.cc:2258] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:43.499765 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 29124
I20260810 18:06:43.500494 30598 ts_tablet_manager.cc:1905] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:43.503849 30598 ts_tablet_manager.cc:1918] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:43.504024 30598 log.cc:1196] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/219b679d06004052bf1e39d7ac6a1468
I20260810 18:06:43.504506 30598 ts_tablet_manager.cc:1939] T 219b679d06004052bf1e39d7ac6a1468 P aba4923d0f9b498289e62debae97813e: Deleting consensus metadata
I20260810 18:06:43.504032 30596 ts_tablet_manager.cc:1905] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:43.509335 30596 ts_tablet_manager.cc:1918] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:43.509473 30596 log.cc:1196] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-0/wal/wals/219b679d06004052bf1e39d7ac6a1468
I20260810 18:06:43.509845 30596 ts_tablet_manager.cc:1939] T 219b679d06004052bf1e39d7ac6a1468 P b9f6e36bb9254c95baa2cc33af78a1fe: Deleting consensus metadata
I20260810 18:06:43.509828 30597 ts_tablet_manager.cc:1905] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:43.512750 30597 ts_tablet_manager.cc:1918] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:43.512894 30597 log.cc:1196] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/219b679d06004052bf1e39d7ac6a1468
I20260810 18:06:43.513293 30597 ts_tablet_manager.cc:1939] T 219b679d06004052bf1e39d7ac6a1468 P 28ec792c34e74d42a3c29be691aac1ca: Deleting consensus metadata
W20260810 18:06:43.514540 30259 connection.cc:535] server connection from 127.0.0.1:58358 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 18:06:43.514832 30413 connection.cc:441] server connection from 127.0.0.1:56358 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56358 (request call id 1) could send its response
W20260810 18:06:43.515070 30104 connection.cc:535] server connection from 127.0.0.1:48398 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 18:06:43.522037 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.498260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.21.182.125:35871)
0810 18:06:43.498762 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.498768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.498785 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:06:43.498831 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.498839 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.498884 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.498988 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:43.504287 (+ 5299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.505137 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.506412 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.506741 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.506863 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.517129 (+ 10266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:43.517134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:43.517139 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:43.517183 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:43.519052 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.519646 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.519653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.519657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.519778 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.520278 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.520284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.520286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.520619 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:43.520743 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.521633 (+ 890us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.521821 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413}
I20260810 18:06:43.530218 30236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.519494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.21.182.124:46311)
0810 18:06:43.519624 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.519628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.520504 (+ 876us) server_negotiation.cc:407] Connection header received
0810 18:06:43.520622 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.520625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.520677 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.520758 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:43.521700 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.522565 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.523987 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.524358 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.524438 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.526333 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:43.526337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:43.526341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:43.526369 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:43.528190 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.528674 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.528678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.528680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.528766 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.529069 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.529075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.529078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.529350 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:43.529496 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.529887 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.530019 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:06:43.532204 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.498415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43348 (local address 127.21.182.124:46311)
0810 18:06:43.498543 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.498548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.521318 (+ 22770us) server_negotiation.cc:407] Connection header received
0810 18:06:43.521426 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.521429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.521489 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.521582 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:43.522999 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.523778 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.525269 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.525634 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.525752 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.528020 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:43.528026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:43.528029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:43.528062 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:43.529887 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.530746 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.530749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.530751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.530840 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.531171 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.531174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.531176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.531389 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:43.531512 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.531841 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.532026 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:06:43.533042 29967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:43.518805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38446 (local address 127.21.182.125:35871)
0810 18:06:43.519306 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:43.519311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:43.519833 (+ 522us) server_negotiation.cc:407] Connection header received
0810 18:06:43.520085 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:43.520089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:43.520148 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:43.520246 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:43.523455 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.524233 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.525367 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:43.525698 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:43.525768 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:43.527818 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:43.527822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:43.527824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:43.527851 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:43.530023 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.530634 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.530638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.530640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.530727 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:43.531382 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:43.531386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:43.531388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:43.531605 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:43.531709 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:43.532762 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0810 18:06:43.532889 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399}
I20260810 18:06:44.477162 30606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:44.476694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.65:39887)
0810 18:06:44.476961 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:44.477053 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
W20260810 18:06:44.477592 30233 heartbeater.cc:644] Failed to heartbeat to 127.21.182.126:37161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
I20260810 18:06:44.478084 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:44.477675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.66:32911)
0810 18:06:44.477924 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:44.478003 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
W20260810 18:06:44.478417 30387 heartbeater.cc:644] Failed to heartbeat to 127.21.182.126:37161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
I20260810 18:06:44.479074 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:44.478684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.67:41371)
0810 18:06:44.478904 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:44.478996 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":61,"threads_started":1}
W20260810 18:06:44.479372 30536 heartbeater.cc:644] Failed to heartbeat to 127.21.182.126:37161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
I20260810 18:06:44.994086 30611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f3ff08bd6f294102b42e2970087bcad4)
I20260810 18:06:44.994086 30610 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f3ff08bd6f294102b42e2970087bcad4)
I20260810 18:06:44.994266 30611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } }
I20260810 18:06:44.994344 30610 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } }
I20260810 18:06:44.994921 30611 leader_election.cc:288] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3ff08bd6f294102b42e2970087bcad4 (127.21.182.126:37161), ea36ca408c3f4a00b71c76b044ca6543 (127.21.182.124:46311)
I20260810 18:06:44.995606 30610 leader_election.cc:288] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3ff08bd6f294102b42e2970087bcad4 (127.21.182.126:37161), 31a294944c674d20ac6295f3ebbd2f9d (127.21.182.125:35871)
I20260810 18:06:44.995791 30613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:44.995151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:34420)
0810 18:06:44.995485 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:44.995589 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":67,"threads_started":1}
W20260810 18:06:44.996032 29814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
I20260810 18:06:44.996575 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:44.995667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:34422)
0810 18:06:44.996363 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:44.996452 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":88,"threads_started":1}
W20260810 18:06:44.996749 30016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
I20260810 18:06:44.997008 30613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:44.996734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:34438)
0810 18:06:44.996883 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:44.996932 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260810 18:06:44.997128 29814 leader_election.cc:334] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f3ff08bd6f294102b42e2970087bcad4 (127.21.182.126:37161): Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
I20260810 18:06:44.997933 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:44.997709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:34452)
0810 18:06:44.997828 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:44.997879 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 18:06:44.998126 30016 leader_election.cc:334] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f3ff08bd6f294102b42e2970087bcad4 (127.21.182.126:37161): Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
I20260810 18:06:45.000326 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:44.996140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.21.182.125:35871)
0810 18:06:44.996392 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:44.996399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:44.996669 (+ 270us) server_negotiation.cc:407] Connection header received
0810 18:06:44.996885 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:44.996891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:44.996933 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:44.996989 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:44.997397 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:44.998143 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:44.999260 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:44.999672 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:44.999808 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:44.999933 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:45.000123 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 18:06:45.000198 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
I20260810 18:06:45.000335 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:44.995320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:46311 (local address 127.0.0.1:43358)
0810 18:06:44.995651 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:44.995668 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:44.996002 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:44.996783 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:44.996787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:44.996809 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:44.996995 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:44.997004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:44.998242 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:44.998245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:44.999124 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:44.999134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:44.999822 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:44.999827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:44.999984 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:45.000007 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:06:45.000104 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 18:06:45.000182 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260810 18:06:45.000349 30615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:44.995350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.21.182.124:46311)
0810 18:06:44.996316 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:44.996322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:44.996338 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:06:44.996392 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:44.996395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:44.996447 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:44.996539 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:44.997262 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:44.998104 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:44.999264 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:44.999687 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:44.999849 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:44.999932 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:45.000120 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 18:06:45.000199 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":882,"thread_start_us":92,"threads_started":1}
I20260810 18:06:45.000356 30617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:44.996070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:35871 (local address 127.0.0.1:38460)
0810 18:06:44.996574 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:44.996588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:44.996757 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:44.997148 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:44.997151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:44.997169 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:44.997272 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:44.997278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:44.998258 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:44.998261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:44.999127 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:44.999137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:44.999802 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:44.999806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:44.999969 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:44.999991 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:06:45.000105 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 18:06:45.000183 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":23,"thread_start_us":67,"threads_started":1}
I20260810 18:06:45.001008 30044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31a294944c674d20ac6295f3ebbd2f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ea36ca408c3f4a00b71c76b044ca6543" is_pre_election: true
I20260810 18:06:45.001307 30044 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a294944c674d20ac6295f3ebbd2f9d in term 1.
I20260810 18:06:45.001467 29833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea36ca408c3f4a00b71c76b044ca6543" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "31a294944c674d20ac6295f3ebbd2f9d" is_pre_election: true
I20260810 18:06:45.001677 29833 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea36ca408c3f4a00b71c76b044ca6543 in term 1.
I20260810 18:06:45.001721 29817 leader_election.cc:302] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a294944c674d20ac6295f3ebbd2f9d, ea36ca408c3f4a00b71c76b044ca6543; no voters: f3ff08bd6f294102b42e2970087bcad4
I20260810 18:06:45.001889 30611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:06:45.001962 30611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 FOLLOWER]: Starting leader election (detected failure of leader f3ff08bd6f294102b42e2970087bcad4)
I20260810 18:06:45.002015 30611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:06:45.002084 30019 leader_election.cc:302] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a294944c674d20ac6295f3ebbd2f9d, ea36ca408c3f4a00b71c76b044ca6543; no voters: f3ff08bd6f294102b42e2970087bcad4
I20260810 18:06:45.002328 30610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:06:45.002400 30610 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f3ff08bd6f294102b42e2970087bcad4)
I20260810 18:06:45.002449 30610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:06:45.003432 30611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } }
I20260810 18:06:45.003757 30610 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } }
I20260810 18:06:45.004220 30610 leader_election.cc:288] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [CANDIDATE]: Term 2 election: Requested vote from peers f3ff08bd6f294102b42e2970087bcad4 (127.21.182.126:37161), 31a294944c674d20ac6295f3ebbd2f9d (127.21.182.125:35871)
I20260810 18:06:45.004086 30611 leader_election.cc:288] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [CANDIDATE]: Term 2 election: Requested vote from peers f3ff08bd6f294102b42e2970087bcad4 (127.21.182.126:37161), ea36ca408c3f4a00b71c76b044ca6543 (127.21.182.124:46311)
I20260810 18:06:45.004309 30044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31a294944c674d20ac6295f3ebbd2f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ea36ca408c3f4a00b71c76b044ca6543"
I20260810 18:06:45.004534 30044 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 31a294944c674d20ac6295f3ebbd2f9d in current term 2: Already voted for candidate ea36ca408c3f4a00b71c76b044ca6543 in this term.
I20260810 18:06:45.004655 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:45.004183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:34458)
0810 18:06:45.004519 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:45.004582 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249}
I20260810 18:06:45.004952 29833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea36ca408c3f4a00b71c76b044ca6543" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "31a294944c674d20ac6295f3ebbd2f9d"
I20260810 18:06:45.005115 29833 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ea36ca408c3f4a00b71c76b044ca6543 in current term 2: Already voted for candidate 31a294944c674d20ac6295f3ebbd2f9d in this term.
I20260810 18:06:45.006125 30613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:45.005925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:34470)
0810 18:06:45.006022 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:45.006071 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 18:06:45.006126 30617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:45.004293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:34462)
0810 18:06:45.005993 (+ 1700us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:45.006051 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1627}
W20260810 18:06:45.006383 29814 leader_election.cc:334] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f3ff08bd6f294102b42e2970087bcad4 (127.21.182.126:37161): Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
I20260810 18:06:45.006448 29814 leader_election.cc:302] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31a294944c674d20ac6295f3ebbd2f9d; no voters: ea36ca408c3f4a00b71c76b044ca6543, f3ff08bd6f294102b42e2970087bcad4
I20260810 18:06:45.006621 30611 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:06:45.006896 30617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:45.006628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:34480)
0810 18:06:45.006787 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:45.006836 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260810 18:06:45.007001 30016 leader_election.cc:334] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f3ff08bd6f294102b42e2970087bcad4 (127.21.182.126:37161): Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
I20260810 18:06:45.007072 30016 leader_election.cc:302] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea36ca408c3f4a00b71c76b044ca6543; no voters: 31a294944c674d20ac6295f3ebbd2f9d, f3ff08bd6f294102b42e2970087bcad4
I20260810 18:06:45.007205 30610 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:06:45.478792 30622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:45.478188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.65:57389)
0810 18:06:45.478573 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:45.478663 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":129,"threads_started":1}
I20260810 18:06:45.479526 30623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:45.479151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.66:50649)
0810 18:06:45.479397 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:45.479449 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260810 18:06:45.480332 30624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:45.479955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.67:53065)
0810 18:06:45.480214 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:45.480258 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
I20260810 18:06:46.480342 30625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:46.479851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.65:47511)
0810 18:06:46.480165 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:46.480238 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":104,"threads_started":1}
I20260810 18:06:46.480719 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:46.480264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.66:55269)
0810 18:06:46.480554 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:46.480621 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":96,"threads_started":1}
I20260810 18:06:46.481539 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:46.481081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.67:48957)
0810 18:06:46.481364 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:46.481432 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":75,"threads_started":1}
I20260810 18:06:47.177719 30629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:47.177940 30629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } }
I20260810 18:06:47.178443 30629 leader_election.cc:288] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f3ff08bd6f294102b42e2970087bcad4 (127.21.182.126:37161), ea36ca408c3f4a00b71c76b044ca6543 (127.21.182.124:46311)
I20260810 18:06:47.179091 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.178652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:34766)
0810 18:06:47.178919 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.179003 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260810 18:06:47.179082 30044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31a294944c674d20ac6295f3ebbd2f9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ea36ca408c3f4a00b71c76b044ca6543" is_pre_election: true
I20260810 18:06:47.179244 30044 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31a294944c674d20ac6295f3ebbd2f9d in term 2.
I20260810 18:06:47.179702 29817 leader_election.cc:302] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a294944c674d20ac6295f3ebbd2f9d, ea36ca408c3f4a00b71c76b044ca6543; no voters:
I20260810 18:06:47.179960 30629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:06:47.180027 30629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:47.180078 30629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:06:47.180088 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.179791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:34772)
0810 18:06:47.179939 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.180002 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 18:06:47.180199 29814 leader_election.cc:334] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f3ff08bd6f294102b42e2970087bcad4 (127.21.182.126:37161): Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
I20260810 18:06:47.183383 30629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } }
I20260810 18:06:47.183946 30629 leader_election.cc:288] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [CANDIDATE]: Term 3 election: Requested vote from peers f3ff08bd6f294102b42e2970087bcad4 (127.21.182.126:37161), ea36ca408c3f4a00b71c76b044ca6543 (127.21.182.124:46311)
I20260810 18:06:47.184398 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.184110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:34798)
0810 18:06:47.184235 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.184313 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 18:06:47.184508 30044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31a294944c674d20ac6295f3ebbd2f9d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ea36ca408c3f4a00b71c76b044ca6543"
I20260810 18:06:47.184671 30044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:06:47.185132 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.184953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:34800)
0810 18:06:47.185034 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.185081 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 18:06:47.185246 29814 leader_election.cc:334] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f3ff08bd6f294102b42e2970087bcad4 (127.21.182.126:37161): Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
I20260810 18:06:47.185866 30044 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31a294944c674d20ac6295f3ebbd2f9d in term 3.
I20260810 18:06:47.186226 29817 leader_election.cc:302] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31a294944c674d20ac6295f3ebbd2f9d, ea36ca408c3f4a00b71c76b044ca6543; no voters: f3ff08bd6f294102b42e2970087bcad4
I20260810 18:06:47.186404 30629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:06:47.186609 30629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [term 3 LEADER]: Becoming Leader. State: Replica: 31a294944c674d20ac6295f3ebbd2f9d, State: Running, Role: LEADER
I20260810 18:06:47.186920 30629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } }
I20260810 18:06:47.188110 30633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31a294944c674d20ac6295f3ebbd2f9d. Latest consensus state: current_term: 3 leader_uuid: "31a294944c674d20ac6295f3ebbd2f9d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } } }
I20260810 18:06:47.188243 30633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:47.188685 30636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:47.191390 30636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:47.191839 30636 catalog_manager.cc:1259] Loaded cluster ID: ca89d301d2e14245aea25dc5cf52a4a8
I20260810 18:06:47.191941 30636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:47.192544 30636 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:47.192958 30636 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: Loaded TSK: 0
I20260810 18:06:47.193707 30636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:06:47.272998 29824 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:06:47.276465 29824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:47.280205 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.279944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:34810)
0810 18:06:47.280060 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.280122 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 18:06:47.280405 30044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [term 3 FOLLOWER]: Refusing update from remote peer 31a294944c674d20ac6295f3ebbd2f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 18:06:47.281142 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.280767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.0.0.1:34820)
0810 18:06:47.280940 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.281017 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 18:06:47.281032 30633 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 18:06:47.281327 29814 consensus_peers.cc:487] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d -> Peer f3ff08bd6f294102b42e2970087bcad4 (127.21.182.126:37161): Couldn't send request to peer f3ff08bd6f294102b42e2970087bcad4. Status: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:06:47.282800 30638 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31a294944c674d20ac6295f3ebbd2f9d. Latest consensus state: current_term: 3 leader_uuid: "31a294944c674d20ac6295f3ebbd2f9d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } } }
I20260810 18:06:47.282954 30638 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:06:47.284525 30633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "31a294944c674d20ac6295f3ebbd2f9d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } } }
I20260810 18:06:47.284667 30633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:47.284600 30638 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "31a294944c674d20ac6295f3ebbd2f9d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3ff08bd6f294102b42e2970087bcad4" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37161 } } peers { permanent_uuid: "31a294944c674d20ac6295f3ebbd2f9d" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 35871 } attrs { promote: false } } peers { permanent_uuid: "ea36ca408c3f4a00b71c76b044ca6543" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 46311 } attrs { promote: false } } }
I20260810 18:06:47.284749 30638 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea36ca408c3f4a00b71c76b044ca6543 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:06:47.297055 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.291810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:36975 (local address 127.0.0.1:58364)
0810 18:06:47.292139 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.292155 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:47.292251 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:47.293296 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:47.293299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:47.293330 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:47.293476 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:47.293484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.294577 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.294580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:47.295771 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:47.295783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.296593 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.296597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:47.296691 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:47.296713 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:06:47.296815 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 18:06:47.296903 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":53,"spinlock_wait_cycles":8832}
I20260810 18:06:47.297056 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.291951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58364 (local address 127.21.182.66:36975)
0810 18:06:47.293043 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:47.293047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:47.293064 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:06:47.293126 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:47.293129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:47.293177 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:47.293286 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:47.293632 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.294444 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.295880 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.296455 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.296582 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:47.296651 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:47.296834 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 18:06:47.296913 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":112,"threads_started":1}
I20260810 18:06:47.297700 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.292179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:42177 (local address 127.0.0.1:48410)
0810 18:06:47.292914 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.292929 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:47.293038 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:47.293452 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:47.293461 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:47.293476 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:47.293553 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:47.293558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.294615 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.294619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:47.296063 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:47.296075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.297237 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.297241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:47.297340 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:47.297365 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:06:47.297442 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 18:06:47.297535 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":30,"thread_start_us":124,"threads_started":1}
I20260810 18:06:47.297710 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.292520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:40943 (local address 127.0.0.1:56366)
0810 18:06:47.293211 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.293227 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:47.293340 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:47.293892 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:47.293895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:47.293925 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:47.294011 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:47.294017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.294973 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.294977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:47.296251 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:47.296260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.297306 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.297310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:47.297416 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:47.297448 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 18:06:47.297524 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 18:06:47.297583 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":1,"spinlock_wait_cycles":7296,"thread_start_us":103,"threads_started":1}
I20260810 18:06:47.297819 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.292641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.21.182.67:40943)
0810 18:06:47.293649 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:47.293654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:47.293670 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:06:47.293720 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:47.293722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:47.293771 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:47.293859 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:47.294143 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.294866 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.296392 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.297085 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.297190 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:47.297254 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:47.297572 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 18:06:47.297674 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":70,"threads_started":1}
I20260810 18:06:47.298147 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.292589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.21.182.65:42177)
0810 18:06:47.292897 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:47.292902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:47.293023 (+ 121us) server_negotiation.cc:407] Connection header received
0810 18:06:47.293141 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:47.293145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:47.293205 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:47.293314 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:47.293683 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.294451 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.296232 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.296953 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.297071 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:47.297154 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:47.297912 (+ 758us) server_negotiation.cc:299] Negotiation successful
0810 18:06:47.298020 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":101,"threads_started":1}
I20260810 18:06:47.298100 30311 tablet_service.cc:1467] Processing CreateTablet for tablet bebe094646e046c7a1dc1cb971282731 (DEFAULT_TABLE table=test-table [id=852dce4da85e4b0f915b485ba7368cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:47.298420 30311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebe094646e046c7a1dc1cb971282731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:47.298699 30158 tablet_service.cc:1467] Processing CreateTablet for tablet bebe094646e046c7a1dc1cb971282731 (DEFAULT_TABLE table=test-table [id=852dce4da85e4b0f915b485ba7368cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:47.298938 30158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebe094646e046c7a1dc1cb971282731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:47.300904 30459 tablet_service.cc:1467] Processing CreateTablet for tablet bebe094646e046c7a1dc1cb971282731 (DEFAULT_TABLE table=test-table [id=852dce4da85e4b0f915b485ba7368cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:47.301229 30459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebe094646e046c7a1dc1cb971282731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:47.301452 30648 tablet_bootstrap.cc:492] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e: Bootstrap starting.
I20260810 18:06:47.301558 30650 tablet_bootstrap.cc:492] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe: Bootstrap starting.
I20260810 18:06:47.302486 30648 tablet_bootstrap.cc:654] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:47.302486 30650 tablet_bootstrap.cc:654] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:47.303681 30654 tablet_bootstrap.cc:492] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca: Bootstrap starting.
I20260810 18:06:47.303886 30650 tablet_bootstrap.cc:492] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe: No bootstrap required, opened a new log
I20260810 18:06:47.303920 30648 tablet_bootstrap.cc:492] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e: No bootstrap required, opened a new log
I20260810 18:06:47.304014 30648 ts_tablet_manager.cc:1397] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:47.304023 30650 ts_tablet_manager.cc:1397] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:47.304487 30648 raft_consensus.cc:348] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:47.304605 30648 raft_consensus.cc:374] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:47.304639 30648 raft_consensus.cc:729] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba4923d0f9b498289e62debae97813e, State: Initialized, Role: FOLLOWER
I20260810 18:06:47.304558 30650 raft_consensus.cc:348] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:47.304682 30650 raft_consensus.cc:374] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:47.304747 30650 raft_consensus.cc:729] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f6e36bb9254c95baa2cc33af78a1fe, State: Initialized, Role: FOLLOWER
I20260810 18:06:47.304769 30648 consensus_queue.cc:260] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:47.304886 30650 consensus_queue.cc:260] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:47.305107 30648 ts_tablet_manager.cc:1428] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:06:47.305259 30650 ts_tablet_manager.cc:1428] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:06:47.305333 30654 tablet_bootstrap.cc:654] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:47.306591 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.306016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.66:46179)
0810 18:06:47.306402 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.306467 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":104,"threads_started":1}
I20260810 18:06:47.306845 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.306335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.65:40035)
0810 18:06:47.306673 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.306738 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":103,"threads_started":1}
I20260810 18:06:47.307451 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.307209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.66:45007)
0810 18:06:47.307335 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.307389 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 18:06:47.310264 30654 tablet_bootstrap.cc:492] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca: No bootstrap required, opened a new log
I20260810 18:06:47.310312 30388 heartbeater.cc:503] Master 127.21.182.125:35871 was elected leader, sending a full tablet report...
I20260810 18:06:47.310374 30654 ts_tablet_manager.cc:1397] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260810 18:06:47.310861 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.309596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.65:55463)
0810 18:06:47.310717 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.310783 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1031}
I20260810 18:06:47.310889 30654 raft_consensus.cc:348] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:47.311014 30654 raft_consensus.cc:374] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:47.311057 30654 raft_consensus.cc:729] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ec792c34e74d42a3c29be691aac1ca, State: Initialized, Role: FOLLOWER
I20260810 18:06:47.311245 30654 consensus_queue.cc:260] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:47.311654 30654 ts_tablet_manager.cc:1428] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 18:06:47.311877 30234 heartbeater.cc:503] Master 127.21.182.125:35871 was elected leader, sending a full tablet report...
I20260810 18:06:47.312829 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.312398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.67:35895)
0810 18:06:47.312659 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.312730 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":114,"threads_started":1}
I20260810 18:06:47.313309 30537 heartbeater.cc:503] Master 127.21.182.125:35871 was elected leader, sending a full tablet report...
I20260810 18:06:47.313898 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.313656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.67:58603)
0810 18:06:47.313771 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.313829 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 18:06:47.531174 30655 raft_consensus.cc:482] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:47.531431 30655 raft_consensus.cc:504] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:47.532997 30655 leader_election.cc:288] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9f6e36bb9254c95baa2cc33af78a1fe (127.21.182.65:42177), 28ec792c34e74d42a3c29be691aac1ca (127.21.182.67:40943)
I20260810 18:06:47.537609 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.533448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:42973 (local address 127.21.182.67:40943)
0810 18:06:47.533647 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:47.533652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:47.533667 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:06:47.533721 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:47.533724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:47.533773 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:47.533898 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:47.534204 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.535009 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.536378 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.536867 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.536987 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:47.537034 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:47.537342 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 18:06:47.537429 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 18:06:47.537616 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.533282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:40943 (local address 127.21.182.66:42973)
0810 18:06:47.533414 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.533432 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:47.533551 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:47.533900 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:47.533904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:47.533928 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:47.534068 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:47.534074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.535140 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.535144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:47.536224 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:47.536235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.537035 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.537039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:47.537156 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:47.537191 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 18:06:47.537268 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 18:06:47.537456 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":28}
I20260810 18:06:47.537863 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.533506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:39739 (local address 127.21.182.65:42177)
0810 18:06:47.533680 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:47.533685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:47.534206 (+ 521us) server_negotiation.cc:407] Connection header received
0810 18:06:47.534306 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:47.534308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:47.534351 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:47.534413 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:47.534734 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.535343 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.536518 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.537092 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.537223 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:47.537288 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:47.537632 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 18:06:47.537731 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 18:06:47.538054 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.533283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:42177 (local address 127.21.182.66:39739)
0810 18:06:47.534060 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.534072 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:47.534217 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:47.534458 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:47.534462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:47.534484 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:47.534611 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:47.534617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.535468 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.535470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:47.536380 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:47.536391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.537349 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.537353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:47.537456 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:47.537479 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:06:47.537569 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 18:06:47.537640 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":370,"thread_start_us":169,"threads_started":1}
I20260810 18:06:47.538331 30480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebe094646e046c7a1dc1cb971282731" candidate_uuid: "aba4923d0f9b498289e62debae97813e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ec792c34e74d42a3c29be691aac1ca" is_pre_election: true
I20260810 18:06:47.538547 30480 raft_consensus.cc:2454] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aba4923d0f9b498289e62debae97813e in term 0.
I20260810 18:06:47.538831 30181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebe094646e046c7a1dc1cb971282731" candidate_uuid: "aba4923d0f9b498289e62debae97813e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" is_pre_election: true
I20260810 18:06:47.539130 30181 raft_consensus.cc:2454] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aba4923d0f9b498289e62debae97813e in term 0.
I20260810 18:06:47.539093 30260 leader_election.cc:302] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28ec792c34e74d42a3c29be691aac1ca, aba4923d0f9b498289e62debae97813e; no voters:
I20260810 18:06:47.539422 30655 raft_consensus.cc:2781] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:47.539515 30655 raft_consensus.cc:482] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:47.539559 30655 raft_consensus.cc:3037] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:47.540772 30655 raft_consensus.cc:504] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:47.541311 30655 leader_election.cc:288] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [CANDIDATE]: Term 1 election: Requested vote from peers b9f6e36bb9254c95baa2cc33af78a1fe (127.21.182.65:42177), 28ec792c34e74d42a3c29be691aac1ca (127.21.182.67:40943)
I20260810 18:06:47.541647 30181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebe094646e046c7a1dc1cb971282731" candidate_uuid: "aba4923d0f9b498289e62debae97813e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe"
I20260810 18:06:47.541790 30181 raft_consensus.cc:3037] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:47.541877 30480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebe094646e046c7a1dc1cb971282731" candidate_uuid: "aba4923d0f9b498289e62debae97813e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28ec792c34e74d42a3c29be691aac1ca"
I20260810 18:06:47.542032 30480 raft_consensus.cc:3037] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:47.542981 30181 raft_consensus.cc:2454] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aba4923d0f9b498289e62debae97813e in term 1.
I20260810 18:06:47.543257 30480 raft_consensus.cc:2454] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aba4923d0f9b498289e62debae97813e in term 1.
I20260810 18:06:47.543363 30258 leader_election.cc:302] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aba4923d0f9b498289e62debae97813e, b9f6e36bb9254c95baa2cc33af78a1fe; no voters:
I20260810 18:06:47.543557 30655 raft_consensus.cc:2781] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:47.543821 30655 raft_consensus.cc:686] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [term 1 LEADER]: Becoming Leader. State: Replica: aba4923d0f9b498289e62debae97813e, State: Running, Role: LEADER
I20260810 18:06:47.544162 30655 consensus_queue.cc:237] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } } peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } }
I20260810 18:06:47.544729 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.544438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.66:36029)
0810 18:06:47.544588 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.544651 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 18:06:47.546926 29824 catalog_manager.cc:5495] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e reported cstate change: term changed from 0 to 1, leader changed from <none> to aba4923d0f9b498289e62debae97813e (127.21.182.66). New cstate: current_term: 1 leader_uuid: "aba4923d0f9b498289e62debae97813e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba4923d0f9b498289e62debae97813e" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 36975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 } health_report { overall_health: UNKNOWN } } }
I20260810 18:06:47.598635 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.594598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.21.182.66:36975)
0810 18:06:47.594741 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:47.594745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:47.594760 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:06:47.594869 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:47.594872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:47.594939 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:47.595041 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:47.595481 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.596119 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.597016 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:47.597238 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:47.597324 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:47.597645 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:47.598201 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:47.598304 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:47.598438 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 18:06:47.598500 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:06:47.602258 30480 raft_consensus.cc:1261] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca [term 1 FOLLOWER]: Refusing update from remote peer aba4923d0f9b498289e62debae97813e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:47.602697 30181 raft_consensus.cc:1261] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe [term 1 FOLLOWER]: Refusing update from remote peer aba4923d0f9b498289e62debae97813e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:47.603031 30664 consensus_queue.cc:1035] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [LEADER]: Connected to new peer: Peer: permanent_uuid: "28ec792c34e74d42a3c29be691aac1ca" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:47.603524 30655 consensus_queue.cc:1035] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9f6e36bb9254c95baa2cc33af78a1fe" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:47.605655 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.605329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.67:47359)
0810 18:06:47.605487 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.605557 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 18:06:47.605839 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.605535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.65:54853)
0810 18:06:47.605710 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.605765 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260810 18:06:47.611387 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.611035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.67:60433)
0810 18:06:47.611236 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.611300 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260810 18:06:47.611538 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.611068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.66:33621)
0810 18:06:47.611386 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.611456 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
I20260810 18:06:47.614589 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.614304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.66:53939)
0810 18:06:47.614456 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.614509 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 18:06:47.615814 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.615434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37161 (local address 127.21.182.65:54681)
0810 18:06:47.615671 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.615726 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260810 18:06:47.622924 29822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:47.623133 29822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:47.627550 29822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31a294944c674d20ac6295f3ebbd2f9d: Sending DeleteTablet for 3 replicas of tablet bebe094646e046c7a1dc1cb971282731
I20260810 18:06:47.628536 30459 tablet_service.cc:1514] Processing DeleteTablet for tablet bebe094646e046c7a1dc1cb971282731 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:47 UTC) from {username='oryx', principal='oryx/127.21.182.125@KRBTEST.COM'} at 127.0.0.1:56366
I20260810 18:06:47.628536 30311 tablet_service.cc:1514] Processing DeleteTablet for tablet bebe094646e046c7a1dc1cb971282731 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:47 UTC) from {username='oryx', principal='oryx/127.21.182.125@KRBTEST.COM'} at 127.0.0.1:58364
I20260810 18:06:47.629371 30681 tablet_replica.cc:330] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca: stopping tablet replica
I20260810 18:06:47.629457 30158 tablet_service.cc:1514] Processing DeleteTablet for tablet bebe094646e046c7a1dc1cb971282731 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:47 UTC) from {username='oryx', principal='oryx/127.21.182.125@KRBTEST.COM'} at 127.0.0.1:48410
I20260810 18:06:47.629526 30681 raft_consensus.cc:2229] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:47.629683 30681 raft_consensus.cc:2258] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:47.629864 30682 tablet_replica.cc:330] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe: stopping tablet replica
I20260810 18:06:47.629997 30682 raft_consensus.cc:2229] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:47.630122 30682 raft_consensus.cc:2258] T bebe094646e046c7a1dc1cb971282731 P b9f6e36bb9254c95baa2cc33af78a1fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:47.630729 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 29991
I20260810 18:06:47.631392 30681 ts_tablet_manager.cc:1905] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:47.635210 30681 ts_tablet_manager.cc:1918] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:47.635301 30681 log.cc:1196] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/bebe094646e046c7a1dc1cb971282731
I20260810 18:06:47.635612 30681 ts_tablet_manager.cc:1939] T bebe094646e046c7a1dc1cb971282731 P 28ec792c34e74d42a3c29be691aac1ca: Deleting consensus metadata
I20260810 18:06:47.636149 30680 tablet_replica.cc:330] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e: stopping tablet replica
I20260810 18:06:47.636323 30680 raft_consensus.cc:2229] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:06:47.636551 29817 catalog_manager.cc:4841] TS 28ec792c34e74d42a3c29be691aac1ca (127.21.182.67:40943): tablet bebe094646e046c7a1dc1cb971282731 (table test-table [id=852dce4da85e4b0f915b485ba7368cb6]) successfully deleted
I20260810 18:06:47.636916 30680 raft_consensus.cc:2258] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:47.638520 30680 ts_tablet_manager.cc:1905] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:47.642064 30680 ts_tablet_manager.cc:1918] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:47.642167 30680 log.cc:1196] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/bebe094646e046c7a1dc1cb971282731
I20260810 18:06:47.642575 30680 ts_tablet_manager.cc:1939] T bebe094646e046c7a1dc1cb971282731 P aba4923d0f9b498289e62debae97813e: Deleting consensus metadata
I20260810 18:06:47.643546 29817 catalog_manager.cc:4841] TS aba4923d0f9b498289e62debae97813e (127.21.182.66:36975): tablet bebe094646e046c7a1dc1cb971282731 (table test-table [id=852dce4da85e4b0f915b485ba7368cb6]) successfully deleted
I20260810 18:06:47.646651 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 30240
I20260810 18:06:47.646683 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:47.646370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:42177 (local address 127.0.0.1:48414)
0810 18:06:47.646520 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:47.646580 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 18:06:47.649600 29815 catalog_manager.cc:4584] TS b9f6e36bb9254c95baa2cc33af78a1fe (127.21.182.65:42177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bebe094646e046c7a1dc1cb971282731: Network error: Client connection negotiation failed: client connection to 127.21.182.65:42177: connect: Connection refused (error 111)
I20260810 18:06:47.656570 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 30393
I20260810 18:06:47.665619 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 29679
I20260810 18:06:47.676383 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 29790
2026-08-10T18:06:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8357 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:06:47 dist-test-slave-27sr krb5kdc[30685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:06:47 dist-test-slave-27sr krb5kdc[30685](info): set up 2 sockets
Aug 10 18:06:47 dist-test-slave-27sr krb5kdc[30685](info): commencing operation
krb5kdc: starting...
W20260810 18:06:48.810393 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:06:48 dist-test-slave-27sr krb5kdc[30685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.126.keytab.
Entry for principal oryx/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.126.keytab.
I20260810 18:06:48.956269 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:40091
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.126.keytab
--principal=oryx/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:40091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:49.079366 30698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:49.079754 30698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:49.079841 30698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:49.084180 30698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:49.084313 30698 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 18:06:49.084347 30698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:49.084391 30698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:49.084430 30698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:49.084467 30698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:49.090255 30698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:40091
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.126.keytab
--principal=oryx/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:40091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:49.091557 30698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:49.092733 30698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:49.102418 30706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:49.102418 30703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:49.102483 30704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:49.103109 30698 server_base.cc:1034] running on GCE node
I20260810 18:06:49.103782 30698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:49.105183 30698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:49.106413 30698 hybrid_clock.cc:648] HybridClock initialized: now 1786385209106400 us; error 31 us; skew 500 ppm
Aug 10 18:06:49 dist-test-slave-27sr krb5kdc[30685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385209, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:49.114465 30698 init.cc:377] Logged in from keytab as oryx/127.21.182.126@KRBTEST.COM (short username oryx)
I20260810 18:06:49.115685 30698 webserver.cc:460] Webserver started at http://127.21.182.126:40471/ using document root <none> and password file <none>
I20260810 18:06:49.116410 30698 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:49.116515 30698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:49.116779 30698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:49.119163 30698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "3b8ae763e7a54263948879b57a7d2923"
format_stamp: "Formatted at 2026-08-10 18:06:49 on dist-test-slave-27sr"
server_key: "3e55a966adcb097a17fbc37c7e112868"
server_key_iv: "7b08f265b8d2ff7340bc772a854fde07"
server_key_version: "encryptionkey@0"
I20260810 18:06:49.119837 30698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "3b8ae763e7a54263948879b57a7d2923"
format_stamp: "Formatted at 2026-08-10 18:06:49 on dist-test-slave-27sr"
server_key: "3e55a966adcb097a17fbc37c7e112868"
server_key_iv: "7b08f265b8d2ff7340bc772a854fde07"
server_key_version: "encryptionkey@0"
I20260810 18:06:49.124253 30698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:06:49.127058 30714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:49.128609 30698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:49.128787 30698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "3b8ae763e7a54263948879b57a7d2923"
format_stamp: "Formatted at 2026-08-10 18:06:49 on dist-test-slave-27sr"
server_key: "3e55a966adcb097a17fbc37c7e112868"
server_key_iv: "7b08f265b8d2ff7340bc772a854fde07"
server_key_version: "encryptionkey@0"
I20260810 18:06:49.128931 30698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:49.146035 30698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:49.149842 30698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:49.150084 30698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:49.159242 30698 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:40091
I20260810 18:06:49.159310 30783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:40091 every 8 connection(s)
I20260810 18:06:49.160405 30698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:49.163836 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:49.165081 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 30698
I20260810 18:06:49.165247 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:49.165632 22233 external_mini_cluster.cc:1428] Setting key 147f834c87e123503dd1e956543b0242
I20260810 18:06:49.170975 30784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923: Bootstrap starting.
I20260810 18:06:49.173949 30784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:49.175020 30784 log.cc:824] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:06:49 dist-test-slave-27sr krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
I20260810 18:06:49.177986 30784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923: No bootstrap required, opened a new log
I20260810 18:06:49.182538 30784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b8ae763e7a54263948879b57a7d2923" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40091 } }
I20260810 18:06:49.182842 30784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:49.182893 30784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8ae763e7a54263948879b57a7d2923, State: Initialized, Role: FOLLOWER
I20260810 18:06:49.183382 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:49.167195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.21.182.126:40091)
0810 18:06:49.167721 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:49.167743 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:49.167785 (+ 42us) server_negotiation.cc:407] Connection header received
0810 18:06:49.168564 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:49.168585 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:49.168971 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:49.169303 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:49.170352 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.171547 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.172875 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.173447 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.173594 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:49.177047 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:49.177055 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:49.177076 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:49.177110 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:49.180022 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:49.180505 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:49.180511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:49.180517 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:49.180664 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:49.180953 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:49.180957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:49.180959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:49.181511 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:49.181728 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:49.182026 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 18:06:49.182358 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":159,"threads_started":1}
I20260810 18:06:49.183408 30784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b8ae763e7a54263948879b57a7d2923" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40091 } }
I20260810 18:06:49.183554 30784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:49.183624 30784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:49.183733 30784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:49.184758 30784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b8ae763e7a54263948879b57a7d2923" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40091 } }
I20260810 18:06:49.185127 30784 leader_election.cc:302] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b8ae763e7a54263948879b57a7d2923; no voters:
I20260810 18:06:49.185499 30784 leader_election.cc:288] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:49.185587 30789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:49.185820 30789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [term 1 LEADER]: Becoming Leader. State: Replica: 3b8ae763e7a54263948879b57a7d2923, State: Running, Role: LEADER
I20260810 18:06:49.186200 30789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b8ae763e7a54263948879b57a7d2923" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40091 } }
I20260810 18:06:49.186928 30784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:49.188256 30790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b8ae763e7a54263948879b57a7d2923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b8ae763e7a54263948879b57a7d2923" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40091 } } }
I20260810 18:06:49.188411 30790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:49.188706 30792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b8ae763e7a54263948879b57a7d2923. Latest consensus state: current_term: 1 leader_uuid: "3b8ae763e7a54263948879b57a7d2923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b8ae763e7a54263948879b57a7d2923" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40091 } } }
I20260810 18:06:49.188874 30792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923 [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:49.188875 30799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:49.192695 30799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:49.198634 30799 catalog_manager.cc:1347] Generated new cluster ID: a4575d3714b14926a14f298c54988dea
I20260810 18:06:49.198747 30799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:49.217669 30799 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:49.218783 30799 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:49.225459 30799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923: Generated new TSK 0
I20260810 18:06:49.226397 30799 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.65.keytab.
Entry for principal oryx/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.65.keytab.
I20260810 18:06:49.267530 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40091
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.65.keytab
--principal=oryx/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:49.387714 30813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:49.388013 30813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:49.388094 30813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:49.392241 30813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:49.392304 30813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:49.392344 30813 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:06:49.392416 30813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:49.397781 30813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.65.keytab
--principal=oryx/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.182.126:40091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:49.398947 30813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:49.400045 30813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:49.411094 30818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:49.411177 30813 server_base.cc:1034] running on GCE node
W20260810 18:06:49.411124 30822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:49.411094 30819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:49.411881 30813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:49.412724 30813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:49.413938 30813 hybrid_clock.cc:648] HybridClock initialized: now 1786385209413919 us; error 39 us; skew 500 ppm
Aug 10 18:06:49 dist-test-slave-27sr krb5kdc[30685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385209, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:49.417106 30813 init.cc:377] Logged in from keytab as oryx/127.21.182.65@KRBTEST.COM (short username oryx)
I20260810 18:06:49.418155 30813 webserver.cc:460] Webserver started at http://127.21.182.65:45229/ using document root <none> and password file <none>
I20260810 18:06:49.418828 30813 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:49.418936 30813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:49.419219 30813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:49.421612 30813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3"
format_stamp: "Formatted at 2026-08-10 18:06:49 on dist-test-slave-27sr"
server_key: "88a43d56f991344d3e6f4bc2d718b0d4"
server_key_iv: "6803303d05d204b69d7d7b062292f845"
server_key_version: "encryptionkey@0"
I20260810 18:06:49.422319 30813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3"
format_stamp: "Formatted at 2026-08-10 18:06:49 on dist-test-slave-27sr"
server_key: "88a43d56f991344d3e6f4bc2d718b0d4"
server_key_iv: "6803303d05d204b69d7d7b062292f845"
server_key_version: "encryptionkey@0"
I20260810 18:06:49.426709 30813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:06:49.429682 30831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:49.430985 30813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:06:49.431166 30813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3"
format_stamp: "Formatted at 2026-08-10 18:06:49 on dist-test-slave-27sr"
server_key: "88a43d56f991344d3e6f4bc2d718b0d4"
server_key_iv: "6803303d05d204b69d7d7b062292f845"
server_key_version: "encryptionkey@0"
I20260810 18:06:49.431299 30813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:49.445209 30813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:49.448300 30813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:49.448522 30813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:49.450016 30813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:49.450093 30813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:49.450179 30813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:49.450227 30813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:06:49 dist-test-slave-27sr krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385209, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
I20260810 18:06:49.461611 30813 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:46387
I20260810 18:06:49.461751 30955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:46387 every 8 connection(s)
I20260810 18:06:49.462874 30813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:49.465229 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:49.451088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:52653 (local address 127.21.182.126:40091)
0810 18:06:49.451220 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:49.451223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:49.452347 (+ 1124us) server_negotiation.cc:407] Connection header received
0810 18:06:49.453560 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:49.453565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:49.453635 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:49.453748 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:49.455378 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.456318 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.457752 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.458063 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.458191 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:49.461034 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:49.461038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:49.461041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:49.461068 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:49.462796 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:49.463581 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:49.463585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:49.463597 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:49.463718 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:49.464161 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:49.464164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:49.464165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:49.464376 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:49.464467 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:49.464986 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 18:06:49.465104 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:06:49.465752 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 30813
I20260810 18:06:49.465910 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:06:49.466099 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:49.451678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40091 (local address 127.21.182.65:52653)
0810 18:06:49.452193 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:49.452236 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:49.453267 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:49.454012 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:49.454021 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:49.454485 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:49.455186 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:49.455206 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.456477 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.456481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:49.457618 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:49.457628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.458242 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.458247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:49.458412 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:49.459004 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:49.459020 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:49.460866 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:49.462996 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:49.463006 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:49.463025 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:49.463433 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:49.463843 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:49.463846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:49.463848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:49.464049 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:49.464476 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:49.464482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:49.464715 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 18:06:49.464867 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 18:06:49.465124 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":166,"threads_started":1}
I20260810 18:06:49.466265 22233 external_mini_cluster.cc:1428] Setting key a28e177cd3bb1e67144561e8fd329afe
I20260810 18:06:49.467583 30956 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40091
I20260810 18:06:49.467798 30956 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:49.468495 30956 heartbeater.cc:511] Master 127.21.182.126:40091 requested a full tablet report, sending...
I20260810 18:06:49.470127 30733 ts_manager.cc:194] Registered new tserver with Master: 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65:46387)
I20260810 18:06:49.471335 30733 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:52653
WARNING: no policy specified for oryx/127.21.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.182.66@KRBTEST.COM" created.
I20260810 18:06:49.481088 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:49.472347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:49197 (local address 127.21.182.126:40091)
0810 18:06:49.472487 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:49.472494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:49.472804 (+ 310us) server_negotiation.cc:407] Connection header received
0810 18:06:49.472943 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:49.472946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:49.472994 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:49.473081 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:49.473686 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.474258 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.475599 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.475813 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.475962 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:49.477349 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:49.477352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:49.477354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:49.477379 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:49.479323 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:49.479768 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:49.479771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:49.479772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:49.479856 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:49.480270 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:49.480273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:49.480274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:49.480472 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:49.480559 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:49.480770 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 18:06:49.480925 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:06:49.481189 30965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:49.472182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40091 (local address 127.21.182.65:49197)
0810 18:06:49.472687 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:49.472705 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:49.472833 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:49.473128 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:49.473133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:49.473397 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:49.473559 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:49.473570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.474408 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.474413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:49.475467 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:49.475483 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.475971 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.475975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:49.476092 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:49.476668 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:49.476688 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:49.477172 (+ 484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:49.479447 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:49.479449 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:49.479451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:49.479649 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:49.480081 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:49.480084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:49.480085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:49.480167 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:49.480563 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:49.480565 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:49.480647 (+ 82us) client_negotiation.cc:769] Sending connection context
0810 18:06:49.480729 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 18:06:49.480912 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.66.keytab.
Entry for principal oryx/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.66.keytab.
I20260810 18:06:49.499689 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40091
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.66.keytab
--principal=oryx/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:49.620321 30967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:49.620585 30967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:49.620630 30967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:49.624585 30967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:49.624648 30967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:49.624688 30967 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:06:49.624720 30967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:49.630244 30967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.66.keytab
--principal=oryx/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.182.126:40091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:49.631264 30967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:49.632323 30967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:49.643046 30975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:49.643055 30972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:49.643103 30973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:49.643299 30967 server_base.cc:1034] running on GCE node
I20260810 18:06:49.643883 30967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:49.644750 30967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:49.645955 30967 hybrid_clock.cc:648] HybridClock initialized: now 1786385209645915 us; error 58 us; skew 500 ppm
Aug 10 18:06:49 dist-test-slave-27sr krb5kdc[30685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385209, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:49.649121 30967 init.cc:377] Logged in from keytab as oryx/127.21.182.66@KRBTEST.COM (short username oryx)
I20260810 18:06:49.650166 30967 webserver.cc:460] Webserver started at http://127.21.182.66:37853/ using document root <none> and password file <none>
I20260810 18:06:49.650813 30967 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:49.650914 30967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:49.651188 30967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:49.653481 30967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "c233d34625464c8ea28439a829d7721a"
format_stamp: "Formatted at 2026-08-10 18:06:49 on dist-test-slave-27sr"
server_key: "07bf2f6cb063acff7a404d315ad15a6f"
server_key_iv: "b31b9ae68e6ebba755a332be256cb0c5"
server_key_version: "encryptionkey@0"
I20260810 18:06:49.654106 30967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "c233d34625464c8ea28439a829d7721a"
format_stamp: "Formatted at 2026-08-10 18:06:49 on dist-test-slave-27sr"
server_key: "07bf2f6cb063acff7a404d315ad15a6f"
server_key_iv: "b31b9ae68e6ebba755a332be256cb0c5"
server_key_version: "encryptionkey@0"
I20260810 18:06:49.658408 30967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:06:49.661168 30983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:49.662518 30967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:06:49.662717 30967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "c233d34625464c8ea28439a829d7721a"
format_stamp: "Formatted at 2026-08-10 18:06:49 on dist-test-slave-27sr"
server_key: "07bf2f6cb063acff7a404d315ad15a6f"
server_key_iv: "b31b9ae68e6ebba755a332be256cb0c5"
server_key_version: "encryptionkey@0"
I20260810 18:06:49.662849 30967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:49.685374 30967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:49.689225 30967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:49.689482 30967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:49.691057 30967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:49.691144 30967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:49.691226 30967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:49.691279 30967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:06:49 dist-test-slave-27sr krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385209, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
I20260810 18:06:49.703253 30967 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:40383
I20260810 18:06:49.703511 31110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:40383 every 8 connection(s)
I20260810 18:06:49.704558 30967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:06:49.706368 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:49.691887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:42803 (local address 127.21.182.126:40091)
0810 18:06:49.692073 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:49.692079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:49.692983 (+ 904us) server_negotiation.cc:407] Connection header received
0810 18:06:49.693938 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:49.693945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:49.694015 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:49.694134 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:49.695655 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.696491 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.697806 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.698040 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.698159 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:49.702189 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:49.702192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:49.702194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:49.702220 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:49.704167 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:49.704726 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:49.704730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:49.704732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:49.704864 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:49.705294 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:49.705298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:49.705299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:49.705524 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:49.705632 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:49.706126 (+ 494us) server_negotiation.cc:299] Negotiation successful
0810 18:06:49.706233 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:06:49.707237 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:49.692333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40091 (local address 127.21.182.66:42803)
0810 18:06:49.692808 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:49.692859 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:49.693713 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:49.694331 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:49.694338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:49.694826 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:49.695488 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:49.695525 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.696618 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.696621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:49.697579 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:49.697592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.698205 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.698209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:49.698429 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:49.699131 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:49.699148 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:49.701978 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:49.704322 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:49.704327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:49.704338 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:49.704619 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:49.704965 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:49.704970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:49.704972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:49.705165 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:49.705621 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:49.705628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:49.705888 (+ 260us) client_negotiation.cc:769] Sending connection context
0810 18:06:49.706063 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 18:06:49.706335 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":133,"threads_started":1}
I20260810 18:06:49.708160 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 30967
I20260810 18:06:49.708312 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:06:49.708617 22233 external_mini_cluster.cc:1428] Setting key 2d9505469a4986d5506a671b70fb7045
I20260810 18:06:49.708885 31111 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40091
I20260810 18:06:49.709044 31111 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:49.709635 31111 heartbeater.cc:511] Master 127.21.182.126:40091 requested a full tablet report, sending...
I20260810 18:06:49.710816 30733 ts_manager.cc:194] Registered new tserver with Master: c233d34625464c8ea28439a829d7721a (127.21.182.66:40383)
I20260810 18:06:49.711625 30733 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:42803
I20260810 18:06:49.720116 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:49.711265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40091 (local address 127.21.182.66:42485)
0810 18:06:49.711634 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:49.711651 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:49.711763 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:49.712186 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:49.712189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:49.712443 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:49.712557 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:49.712565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.713734 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.713737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:49.714467 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:49.714480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.715042 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.715048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:49.715162 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:49.715777 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:49.715790 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:49.716228 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:49.718272 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:49.718275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:49.718276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:49.718490 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:49.718842 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:49.718845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:49.718846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:49.718930 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:49.719518 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:49.719520 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:49.719609 (+ 89us) client_negotiation.cc:769] Sending connection context
0810 18:06:49.719773 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 18:06:49.719963 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":90,"threads_started":1}
I20260810 18:06:49.720207 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:49.711304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:42485 (local address 127.21.182.126:40091)
0810 18:06:49.711426 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:49.711430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:49.711722 (+ 292us) server_negotiation.cc:407] Connection header received
0810 18:06:49.711871 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:49.711874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:49.712079 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:49.712165 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:49.712698 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.713575 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.714601 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.714917 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.715011 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:49.716371 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:49.716377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:49.716383 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:49.716416 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:49.718147 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:49.718600 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:49.718606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:49.718611 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:49.718727 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:49.719031 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:49.719038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:49.719041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:49.719397 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:49.719519 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:49.719854 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 18:06:49.720017 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for oryx/127.21.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.67.keytab.
Entry for principal oryx/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.67.keytab.
I20260810 18:06:49.743777 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40091
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.67.keytab
--principal=oryx/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:49.864166 31120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:49.864600 31120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:49.864734 31120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:49.869645 31120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:49.869800 31120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:49.869843 31120 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:06:49.869877 31120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:49.875340 31120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.67.keytab
--principal=oryx/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.182.126:40091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:49.876521 31120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:49.877544 31120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:49.887674 31128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:49.887804 31127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:49.887985 31120 server_base.cc:1034] running on GCE node
W20260810 18:06:49.888010 31130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:49.888541 31120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:49.889387 31120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:49.890625 31120 hybrid_clock.cc:648] HybridClock initialized: now 1786385209890579 us; error 54 us; skew 500 ppm
Aug 10 18:06:49 dist-test-slave-27sr krb5kdc[30685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385209, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:49.898564 31120 init.cc:377] Logged in from keytab as oryx/127.21.182.67@KRBTEST.COM (short username oryx)
I20260810 18:06:49.899768 31120 webserver.cc:460] Webserver started at http://127.21.182.67:38575/ using document root <none> and password file <none>
I20260810 18:06:49.900487 31120 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:49.900592 31120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:49.900848 31120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:49.903177 31120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "38a88b307b7545b5a622c9df9ad2aed5"
format_stamp: "Formatted at 2026-08-10 18:06:49 on dist-test-slave-27sr"
server_key: "fe84811498a653a3b2b9db83a9730b3e"
server_key_iv: "dba36b8d38c7cdbac1eaae2f67c73410"
server_key_version: "encryptionkey@0"
I20260810 18:06:49.903823 31120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "38a88b307b7545b5a622c9df9ad2aed5"
format_stamp: "Formatted at 2026-08-10 18:06:49 on dist-test-slave-27sr"
server_key: "fe84811498a653a3b2b9db83a9730b3e"
server_key_iv: "dba36b8d38c7cdbac1eaae2f67c73410"
server_key_version: "encryptionkey@0"
I20260810 18:06:49.908028 31120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:06:49.910897 31138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:49.912498 31120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:49.912623 31120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "38a88b307b7545b5a622c9df9ad2aed5"
format_stamp: "Formatted at 2026-08-10 18:06:49 on dist-test-slave-27sr"
server_key: "fe84811498a653a3b2b9db83a9730b3e"
server_key_iv: "dba36b8d38c7cdbac1eaae2f67c73410"
server_key_version: "encryptionkey@0"
I20260810 18:06:49.912755 31120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:49.946884 31120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:49.952351 31120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:49.952602 31120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:49.954825 31120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:49.954914 31120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:49.954988 31120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:49.955027 31120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:06:49 dist-test-slave-27sr krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385209, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
I20260810 18:06:49.969745 31120 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:33069
I20260810 18:06:49.969838 31272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:33069 every 8 connection(s)
I20260810 18:06:49.970784 31120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:06:49.973552 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:49.955966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:55259 (local address 127.21.182.126:40091)
0810 18:06:49.956155 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:49.956160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:49.957263 (+ 1103us) server_negotiation.cc:407] Connection header received
0810 18:06:49.958755 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:49.958760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:49.958838 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:49.958909 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:49.961078 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.961922 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.964168 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.964619 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.964786 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:49.968746 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:49.968755 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:49.968757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:49.968790 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:49.971008 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:49.971781 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:49.971784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:49.971785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:49.971883 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:49.972308 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:49.972311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:49.972312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:49.972528 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:49.972608 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:49.973273 (+ 665us) server_negotiation.cc:299] Negotiation successful
0810 18:06:49.973415 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:06:49.974421 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 31120
I20260810 18:06:49.974490 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:49.956522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40091 (local address 127.21.182.67:55259)
0810 18:06:49.957068 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:49.957127 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:49.958416 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:49.959436 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:49.959447 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:49.960061 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:49.960896 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:49.960923 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.962091 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.962097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:49.963823 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:49.963836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.964866 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.964871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:49.965131 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:49.965959 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:49.965985 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:49.968480 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:49.971162 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:49.971170 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:49.971188 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:49.971621 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:49.972041 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:49.972045 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:49.972046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:49.972202 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:49.972656 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:49.972662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:49.972919 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 18:06:49.973085 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 18:06:49.973370 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":153,"threads_started":1}
I20260810 18:06:49.974594 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:06:49.974942 22233 external_mini_cluster.cc:1428] Setting key d4aeab3eb28c79899893f1a983592114
I20260810 18:06:49.976679 31273 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40091
I20260810 18:06:49.976868 31273 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:49.977545 31273 heartbeater.cc:511] Master 127.21.182.126:40091 requested a full tablet report, sending...
I20260810 18:06:49.978669 30735 ts_manager.cc:194] Registered new tserver with Master: 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67:33069)
I20260810 18:06:49.979544 30735 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:55259
I20260810 18:06:49.989078 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:49.978947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40091 (local address 127.21.182.67:46723)
0810 18:06:49.979303 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:49.979317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:49.979418 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:49.979666 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:49.979669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:49.979986 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:49.980095 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:49.980101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.982210 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.982214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:49.982815 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:49.982824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.983349 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.983352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:49.983609 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:49.984279 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:49.984293 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:49.984781 (+ 488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:49.987167 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:49.987170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:49.987172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:49.987438 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:49.987756 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:49.987760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:49.987762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:49.987875 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:49.988454 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:49.988456 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:49.988541 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 18:06:49.988661 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 18:06:49.988858 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":143,"threads_started":1}
I20260810 18:06:49.989248 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:49.979013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:46723 (local address 127.21.182.126:40091)
0810 18:06:49.979160 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:49.979165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:49.979394 (+ 229us) server_negotiation.cc:407] Connection header received
0810 18:06:49.979505 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:49.979507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:49.979548 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:49.979615 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:49.981555 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.982067 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.982948 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:49.983177 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:49.983342 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:49.984971 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:49.984975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:49.984977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:49.985005 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:49.987022 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:49.987557 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:49.987560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:49.987561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:49.987649 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:49.988045 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:49.988048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:49.988050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:49.988318 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:49.988386 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:49.988852 (+ 466us) server_negotiation.cc:299] Negotiation successful
0810 18:06:49.989032 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:06:49.990311 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:06:49 dist-test-slave-27sr krb5kdc[30685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385209, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:06:50 dist-test-slave-27sr krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:50 dist-test-slave-27sr krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:06:50.018469 30787 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:50.011456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.21.182.126:40091)
0810 18:06:50.011598 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.011601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.011614 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:06:50.011662 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.011664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.011707 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.011808 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:50.012633 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.013186 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.014121 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.014352 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.014466 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.018300 (+ 3834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43140: BlockingRecv error: recv got EOF from 127.0.0.1:43140 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 18:06:50 dist-test-slave-27sr krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385209, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
I20260810 18:06:50.029193 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.020157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.21.182.126:40091)
0810 18:06:50.020318 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.020323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.020425 (+ 102us) server_negotiation.cc:407] Connection header received
0810 18:06:50.020609 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.020611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.020656 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.020745 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:50.021393 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.021959 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.022861 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.023087 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.023199 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.025788 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:50.025791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:50.025794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:50.025824 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:50.027201 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:50.027622 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:50.027625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:50.027626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:50.027706 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:50.028140 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:50.028142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:50.028143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:50.028366 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:50.028480 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.028932 (+ 452us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.029061 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:06:50.033079 30735 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:06:50.036988 30735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:50.056432 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.049086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:46387 (local address 127.0.0.1:60452)
0810 18:06:50.049923 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:50.049988 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:50.050137 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:50.050999 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:50.051007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:50.051038 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:50.051232 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.051240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.052834 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.052840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.054364 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.054376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.055672 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.055679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.055862 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.056005 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 18:06:50.056119 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 18:06:50.056264 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":57,"spinlock_wait_cycles":2688,"thread_start_us":93,"threads_started":1}
I20260810 18:06:50.056545 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.049199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.21.182.65:46387)
0810 18:06:50.050445 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.050452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.050474 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:06:50.050572 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.050577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.050830 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.050957 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:50.051440 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.052690 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.054539 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.055505 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.055958 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.056164 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.056224 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.056353 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1137,"thread_start_us":91,"threads_started":1}
I20260810 18:06:50.057452 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.049521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:33069 (local address 127.0.0.1:46250)
0810 18:06:50.049923 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:50.049988 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:50.050137 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:50.052891 (+ 2754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:50.052893 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:50.052906 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:50.053056 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.053062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.054565 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.054567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.055660 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.055674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.057081 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.057084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.057173 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.057197 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:06:50.057247 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 18:06:50.057316 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":54,"spinlock_wait_cycles":6912,"thread_start_us":141,"threads_started":1}
I20260810 18:06:50.058202 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.052014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.21.182.67:33069)
0810 18:06:50.052464 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.052474 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.052493 (+ 19us) server_negotiation.cc:407] Connection header received
0810 18:06:50.052564 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.052569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.052745 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.053202 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:50.053371 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.054447 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.056127 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.056941 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.057740 (+ 799us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.057914 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.057967 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.058052 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":121,"threads_started":1}
I20260810 18:06:50.060534 31300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.049878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:40383 (local address 127.0.0.1:55408)
0810 18:06:50.052050 (+ 2172us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:50.052064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:50.052163 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:50.052853 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:50.052857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:50.052880 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:50.053055 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.053062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.054539 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.054545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.058552 (+ 4007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.058563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.059915 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.059921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.060032 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.060057 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:06:50.060130 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 18:06:50.060373 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2106,"mutex_wait_us":28,"thread_start_us":96,"threads_started":1}
I20260810 18:06:50.061173 31301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.050349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55408 (local address 127.21.182.66:40383)
0810 18:06:50.050726 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.050737 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.052235 (+ 1498us) server_negotiation.cc:407] Connection header received
0810 18:06:50.052329 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.052334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.052533 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.052688 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:50.053190 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.054388 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.058730 (+ 4342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.059559 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.059669 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.059825 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.060863 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.060978 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":143,"threads_started":1}
I20260810 18:06:50.062242 31197 tablet_service.cc:1467] Processing CreateTablet for tablet c4fa7c5df121441d93bec8276234b717 (DEFAULT_TABLE table=test-table [id=accb150bd21744baae941046d0300b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:50.063632 31197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4fa7c5df121441d93bec8276234b717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.065264 30885 tablet_service.cc:1467] Processing CreateTablet for tablet c4fa7c5df121441d93bec8276234b717 (DEFAULT_TABLE table=test-table [id=accb150bd21744baae941046d0300b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:50.066536 30885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4fa7c5df121441d93bec8276234b717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.067319 31040 tablet_service.cc:1467] Processing CreateTablet for tablet c4fa7c5df121441d93bec8276234b717 (DEFAULT_TABLE table=test-table [id=accb150bd21744baae941046d0300b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:50.069573 31040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4fa7c5df121441d93bec8276234b717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.074802 31305 tablet_bootstrap.cc:492] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Bootstrap starting.
I20260810 18:06:50.075589 31304 tablet_bootstrap.cc:492] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5: Bootstrap starting.
I20260810 18:06:50.077497 31305 tablet_bootstrap.cc:654] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.078027 31304 tablet_bootstrap.cc:654] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.078495 31305 log.cc:824] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:50.078991 31304 log.cc:824] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:50.080811 31305 tablet_bootstrap.cc:492] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: No bootstrap required, opened a new log
I20260810 18:06:50.081018 31305 ts_tablet_manager.cc:1397] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260810 18:06:50.081427 31304 tablet_bootstrap.cc:492] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5: No bootstrap required, opened a new log
I20260810 18:06:50.081601 31304 ts_tablet_manager.cc:1397] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:06:50.085831 31304 raft_consensus.cc:348] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.085829 31305 raft_consensus.cc:348] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.086119 31305 raft_consensus.cc:374] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.086119 31304 raft_consensus.cc:374] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.086174 31304 raft_consensus.cc:729] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a88b307b7545b5a622c9df9ad2aed5, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.086174 31305 raft_consensus.cc:729] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c47f0fd74c4923b7a1ba3dd7e41ae3, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.086678 31304 consensus_queue.cc:260] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.086678 31305 consensus_queue.cc:260] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.087406 30956 heartbeater.cc:503] Master 127.21.182.126:40091 was elected leader, sending a full tablet report...
I20260810 18:06:50.087406 31273 heartbeater.cc:503] Master 127.21.182.126:40091 was elected leader, sending a full tablet report...
I20260810 18:06:50.087875 31304 ts_tablet_manager.cc:1428] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260810 18:06:50.088187 31305 ts_tablet_manager.cc:1428] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 18:06:50.088954 31307 tablet_bootstrap.cc:492] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a: Bootstrap starting.
I20260810 18:06:50.092024 31307 tablet_bootstrap.cc:654] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.093124 31307 log.cc:824] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:50.095957 31307 tablet_bootstrap.cc:492] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a: No bootstrap required, opened a new log
I20260810 18:06:50.096226 31307 ts_tablet_manager.cc:1397] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260810 18:06:50.101750 31307 raft_consensus.cc:348] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.102120 31307 raft_consensus.cc:374] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.102195 31307 raft_consensus.cc:729] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c233d34625464c8ea28439a829d7721a, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.102782 31307 consensus_queue.cc:260] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.103690 31111 heartbeater.cc:503] Master 127.21.182.126:40091 was elected leader, sending a full tablet report...
I20260810 18:06:50.104197 31307 ts_tablet_manager.cc:1428] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260810 18:06:50.135993 31310 raft_consensus.cc:482] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:50.136402 31310 raft_consensus.cc:504] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.138105 31310 leader_election.cc:288] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65:46387), c233d34625464c8ea28439a829d7721a (127.21.182.66:40383)
I20260810 18:06:50.141124 31309 raft_consensus.cc:482] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:50.141345 31309 raft_consensus.cc:504] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.142709 31309 leader_election.cc:288] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c233d34625464c8ea28439a829d7721a (127.21.182.66:40383), 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67:33069)
I20260810 18:06:50.142901 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.138511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:46387 (local address 127.21.182.67:41525)
0810 18:06:50.138691 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:50.138706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:50.138811 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:50.139292 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:50.139296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:50.139330 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:50.139480 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.139490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.140542 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.140548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.141615 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.141625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.142435 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.142438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.142553 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.142581 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:06:50.142661 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 18:06:50.142738 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 18:06:50.143358 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.138667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:41525 (local address 127.21.182.65:46387)
0810 18:06:50.139036 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.139042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.139058 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:06:50.139119 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.139122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.139174 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.139259 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:50.139607 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.140405 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.141746 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.142300 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.142427 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.142479 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.143169 (+ 690us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.143251 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266}
I20260810 18:06:50.143396 31301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.139083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:33039 (local address 127.21.182.66:40383)
0810 18:06:50.139226 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.139232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.139378 (+ 146us) server_negotiation.cc:407] Connection header received
0810 18:06:50.139486 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.139488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.139527 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.139631 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:50.140394 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.140947 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.142133 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.142699 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.142801 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.142867 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.143203 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.143278 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:06:50.143771 31313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.138866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:40383 (local address 127.21.182.67:33039)
0810 18:06:50.139299 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:50.139312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:50.139403 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:50.139655 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:50.139657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:50.139672 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:50.139763 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.139766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.141067 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.141070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.142017 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.142025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.142945 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.142949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.143050 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.143080 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 18:06:50.143560 (+ 480us) client_negotiation.cc:240] Negotiation successful
0810 18:06:50.143643 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":168,"threads_started":1}
I20260810 18:06:50.144089 30908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4fa7c5df121441d93bec8276234b717" candidate_uuid: "38a88b307b7545b5a622c9df9ad2aed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" is_pre_election: true
I20260810 18:06:50.144323 30908 raft_consensus.cc:2454] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38a88b307b7545b5a622c9df9ad2aed5 in term 0.
I20260810 18:06:50.144855 31140 leader_election.cc:302] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c47f0fd74c4923b7a1ba3dd7e41ae3, 38a88b307b7545b5a622c9df9ad2aed5; no voters:
I20260810 18:06:50.145131 31310 raft_consensus.cc:2781] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:50.145223 31310 raft_consensus.cc:482] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:50.145272 31310 raft_consensus.cc:3037] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:50.145985 31063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4fa7c5df121441d93bec8276234b717" candidate_uuid: "38a88b307b7545b5a622c9df9ad2aed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c233d34625464c8ea28439a829d7721a" is_pre_election: true
I20260810 18:06:50.146293 31063 raft_consensus.cc:2454] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38a88b307b7545b5a622c9df9ad2aed5 in term 0.
I20260810 18:06:50.146631 31310 raft_consensus.cc:504] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.147006 31310 leader_election.cc:288] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [CANDIDATE]: Term 1 election: Requested vote from peers 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65:46387), c233d34625464c8ea28439a829d7721a (127.21.182.66:40383)
I20260810 18:06:50.147377 30908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4fa7c5df121441d93bec8276234b717" candidate_uuid: "38a88b307b7545b5a622c9df9ad2aed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3"
I20260810 18:06:50.147509 30908 raft_consensus.cc:3037] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:50.148097 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.143372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:48973 (local address 127.21.182.67:33069)
0810 18:06:50.143491 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.143495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.143933 (+ 438us) server_negotiation.cc:407] Connection header received
0810 18:06:50.144105 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.144111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.144156 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.144230 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:50.144612 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.145343 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.146862 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.147423 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.147546 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.147611 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.147924 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.147990 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:06:50.148149 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.143030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:33069 (local address 127.21.182.65:48973)
0810 18:06:50.143826 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:50.143842 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:50.143993 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:50.144336 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:50.144341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:50.144364 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:50.144466 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.144473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.145444 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.145447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.146699 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.146710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.147679 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.147683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.147762 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.147779 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 18:06:50.147874 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 18:06:50.147973 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":25,"thread_start_us":100,"threads_started":1}
I20260810 18:06:50.148689 31301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.143550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:33055 (local address 127.21.182.66:40383)
0810 18:06:50.143673 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.143677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.144234 (+ 557us) server_negotiation.cc:407] Connection header received
0810 18:06:50.144536 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.144539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.144587 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.144661 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:50.144923 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.145616 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.147277 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.147953 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.148043 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.148091 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.148551 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.148609 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:06:50.148823 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.142857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:40383 (local address 127.21.182.65:33055)
0810 18:06:50.144010 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:50.144024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:50.144348 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:50.144674 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:50.144679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:50.144698 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:50.144799 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.144805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.145734 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.145738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.147140 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.147153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.148314 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.148317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.148412 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.148436 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:06:50.148517 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 18:06:50.148582 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":30,"thread_start_us":118,"threads_started":1}
I20260810 18:06:50.148973 30908 raft_consensus.cc:2454] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38a88b307b7545b5a622c9df9ad2aed5 in term 1.
I20260810 18:06:50.149674 31063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4fa7c5df121441d93bec8276234b717" candidate_uuid: "38a88b307b7545b5a622c9df9ad2aed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c233d34625464c8ea28439a829d7721a"
I20260810 18:06:50.149677 31218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4fa7c5df121441d93bec8276234b717" candidate_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a88b307b7545b5a622c9df9ad2aed5" is_pre_election: true
I20260810 18:06:50.149737 31061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4fa7c5df121441d93bec8276234b717" candidate_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c233d34625464c8ea28439a829d7721a" is_pre_election: true
I20260810 18:06:50.149806 31063 raft_consensus.cc:3037] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:50.149916 31218 raft_consensus.cc:2379] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02c47f0fd74c4923b7a1ba3dd7e41ae3 in current term 1: Already voted for candidate 38a88b307b7545b5a622c9df9ad2aed5 in this term.
I20260810 18:06:50.151472 31063 raft_consensus.cc:2454] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38a88b307b7545b5a622c9df9ad2aed5 in term 1.
I20260810 18:06:50.151774 30833 leader_election.cc:302] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02c47f0fd74c4923b7a1ba3dd7e41ae3; no voters: 38a88b307b7545b5a622c9df9ad2aed5, c233d34625464c8ea28439a829d7721a
I20260810 18:06:50.151813 31140 leader_election.cc:302] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a88b307b7545b5a622c9df9ad2aed5, c233d34625464c8ea28439a829d7721a; no voters:
I20260810 18:06:50.152135 31310 raft_consensus.cc:2781] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:50.152120 31309 raft_consensus.cc:2726] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:06:50.152388 31310 raft_consensus.cc:686] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 LEADER]: Becoming Leader. State: Replica: 38a88b307b7545b5a622c9df9ad2aed5, State: Running, Role: LEADER
I20260810 18:06:50.152731 31310 consensus_queue.cc:237] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.156111 30735 catalog_manager.cc:5495] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67). New cstate: current_term: 1 leader_uuid: "38a88b307b7545b5a622c9df9ad2aed5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } health_report { overall_health: HEALTHY } } }
I20260810 18:06:50.182850 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.179375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46260 (local address 127.21.182.67:33069)
0810 18:06:50.179515 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.179518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.179533 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:06:50.179614 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.179616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.179671 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.179824 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:50.180099 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.180599 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.181430 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.181642 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.181721 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.182010 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:50.182472 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:50.182565 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.182623 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.182672 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:06:50.189050 31063 raft_consensus.cc:1261] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Refusing update from remote peer 38a88b307b7545b5a622c9df9ad2aed5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:50.189095 30908 raft_consensus.cc:1261] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Refusing update from remote peer 38a88b307b7545b5a622c9df9ad2aed5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:50.189750 31316 consensus_queue.cc:1035] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:50.189972 31310 consensus_queue.cc:1035] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:50.198596 31321 mvcc.cc:205] Tried to move back new op lower bound from 7317033820926197760 to 7317033820788580352. Current Snapshot: MvccSnapshot[applied={T|T < 7317033820926197760}]
I20260810 18:06:50.198882 31320 mvcc.cc:205] Tried to move back new op lower bound from 7317033820926197760 to 7317033820788580352. Current Snapshot: MvccSnapshot[applied={T|T < 7317033820926197760}]
I20260810 18:06:50.207376 30733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:50.207582 30733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:50.210395 30733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923: Sending DeleteTablet for 3 replicas of tablet c4fa7c5df121441d93bec8276234b717
I20260810 18:06:50.211292 30885 tablet_service.cc:1514] Processing DeleteTablet for tablet c4fa7c5df121441d93bec8276234b717 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:50 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:60452
I20260810 18:06:50.211380 31040 tablet_service.cc:1514] Processing DeleteTablet for tablet c4fa7c5df121441d93bec8276234b717 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:50 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:55408
I20260810 18:06:50.211835 31197 tablet_service.cc:1514] Processing DeleteTablet for tablet c4fa7c5df121441d93bec8276234b717 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:50 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:46250
I20260810 18:06:50.212096 31329 tablet_replica.cc:330] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a: stopping tablet replica
I20260810 18:06:50.212409 30733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 18:06:50.212492 31329 raft_consensus.cc:2229] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 18:06:50.212867 30733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:50.212854 31329 raft_consensus.cc:2258] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:50.213246 31331 tablet_replica.cc:330] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: stopping tablet replica
I20260810 18:06:50.213582 31331 raft_consensus.cc:2229] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:50.213794 31332 tablet_replica.cc:330] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5: stopping tablet replica
I20260810 18:06:50.213897 31331 raft_consensus.cc:2258] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:50.214059 31332 raft_consensus.cc:2229] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:06:50.214438 31332 raft_consensus.cc:2258] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:50.216603 31331 ts_tablet_manager.cc:1905] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:50.218255 31040 tablet_service.cc:1467] Processing CreateTablet for tablet 014b99bdba6f41fcb50c0c7eb9171ec4 (DEFAULT_TABLE table=test-table [id=dac868aa539d41e0aa89524fc86f2877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:50.218561 31040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014b99bdba6f41fcb50c0c7eb9171ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.218735 30885 tablet_service.cc:1467] Processing CreateTablet for tablet 014b99bdba6f41fcb50c0c7eb9171ec4 (DEFAULT_TABLE table=test-table [id=dac868aa539d41e0aa89524fc86f2877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:50.219004 30885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014b99bdba6f41fcb50c0c7eb9171ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.219051 31329 ts_tablet_manager.cc:1905] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:50.219883 31332 ts_tablet_manager.cc:1905] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:50.221487 31305 tablet_bootstrap.cc:492] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Bootstrap starting.
I20260810 18:06:50.221944 31307 tablet_bootstrap.cc:492] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a: Bootstrap starting.
I20260810 18:06:50.221961 31197 tablet_service.cc:1467] Processing CreateTablet for tablet 014b99bdba6f41fcb50c0c7eb9171ec4 (DEFAULT_TABLE table=test-table [id=dac868aa539d41e0aa89524fc86f2877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:50.222546 31305 tablet_bootstrap.cc:654] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.222892 31307 tablet_bootstrap.cc:654] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.223867 31305 tablet_bootstrap.cc:492] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: No bootstrap required, opened a new log
I20260810 18:06:50.224000 31305 ts_tablet_manager.cc:1397] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:50.224191 31307 tablet_bootstrap.cc:492] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a: No bootstrap required, opened a new log
I20260810 18:06:50.224295 31307 ts_tablet_manager.cc:1397] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 18:06:50.224515 31305 raft_consensus.cc:348] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.224646 31305 raft_consensus.cc:374] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.224687 31305 raft_consensus.cc:729] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c47f0fd74c4923b7a1ba3dd7e41ae3, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.224767 31307 raft_consensus.cc:348] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.224888 31307 raft_consensus.cc:374] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.224864 31305 consensus_queue.cc:260] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.224936 31307 raft_consensus.cc:729] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c233d34625464c8ea28439a829d7721a, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.225092 31307 consensus_queue.cc:260] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.225445 31197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014b99bdba6f41fcb50c0c7eb9171ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.225814 31307 ts_tablet_manager.cc:1428] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:06:50.226055 31331 ts_tablet_manager.cc:1918] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:50.226158 31331 log.cc:1196] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/wal/wals/c4fa7c5df121441d93bec8276234b717
I20260810 18:06:50.226686 31331 ts_tablet_manager.cc:1939] T c4fa7c5df121441d93bec8276234b717 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Deleting consensus metadata
I20260810 18:06:50.228128 31304 tablet_bootstrap.cc:492] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5: Bootstrap starting.
I20260810 18:06:50.228335 30718 catalog_manager.cc:4841] TS 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65:46387): tablet c4fa7c5df121441d93bec8276234b717 (table test-table [id=accb150bd21744baae941046d0300b11]) successfully deleted
I20260810 18:06:50.229193 31304 tablet_bootstrap.cc:654] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.229671 31305 ts_tablet_manager.cc:1428] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260810 18:06:50.231117 31304 tablet_bootstrap.cc:492] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5: No bootstrap required, opened a new log
I20260810 18:06:50.231230 31304 ts_tablet_manager.cc:1397] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 18:06:50.231694 31304 raft_consensus.cc:348] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.231824 31304 raft_consensus.cc:374] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.231856 31304 raft_consensus.cc:729] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a88b307b7545b5a622c9df9ad2aed5, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.232015 31304 consensus_queue.cc:260] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.232458 31304 ts_tablet_manager.cc:1428] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260810 18:06:50.232681 31332 ts_tablet_manager.cc:1918] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:50.233065 31329 ts_tablet_manager.cc:1918] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:50.233106 31332 log.cc:1196] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/c4fa7c5df121441d93bec8276234b717
I20260810 18:06:50.233179 31329 log.cc:1196] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/c4fa7c5df121441d93bec8276234b717
I20260810 18:06:50.233479 31332 ts_tablet_manager.cc:1939] T c4fa7c5df121441d93bec8276234b717 P 38a88b307b7545b5a622c9df9ad2aed5: Deleting consensus metadata
I20260810 18:06:50.233582 31329 ts_tablet_manager.cc:1939] T c4fa7c5df121441d93bec8276234b717 P c233d34625464c8ea28439a829d7721a: Deleting consensus metadata
I20260810 18:06:50.234809 30720 catalog_manager.cc:4841] TS 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67:33069): tablet c4fa7c5df121441d93bec8276234b717 (table test-table [id=accb150bd21744baae941046d0300b11]) successfully deleted
I20260810 18:06:50.235107 30718 catalog_manager.cc:4841] TS c233d34625464c8ea28439a829d7721a (127.21.182.66:40383): tablet c4fa7c5df121441d93bec8276234b717 (table test-table [id=accb150bd21744baae941046d0300b11]) successfully deleted
I20260810 18:06:50.241783 31325 raft_consensus.cc:482] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:50.241917 31325 raft_consensus.cc:504] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.242357 31325 leader_election.cc:288] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65:46387), c233d34625464c8ea28439a829d7721a (127.21.182.66:40383)
I20260810 18:06:50.242791 30908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014b99bdba6f41fcb50c0c7eb9171ec4" candidate_uuid: "38a88b307b7545b5a622c9df9ad2aed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" is_pre_election: true
I20260810 18:06:50.242857 31063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014b99bdba6f41fcb50c0c7eb9171ec4" candidate_uuid: "38a88b307b7545b5a622c9df9ad2aed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c233d34625464c8ea28439a829d7721a" is_pre_election: true
I20260810 18:06:50.242983 30908 raft_consensus.cc:2454] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38a88b307b7545b5a622c9df9ad2aed5 in term 0.
I20260810 18:06:50.243013 31063 raft_consensus.cc:2454] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38a88b307b7545b5a622c9df9ad2aed5 in term 0.
I20260810 18:06:50.243304 31140 leader_election.cc:302] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a88b307b7545b5a622c9df9ad2aed5, c233d34625464c8ea28439a829d7721a; no voters:
I20260810 18:06:50.243589 31325 raft_consensus.cc:2781] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:50.243640 31325 raft_consensus.cc:482] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:50.243666 31325 raft_consensus.cc:3037] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:50.244760 31325 raft_consensus.cc:504] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.245147 31325 leader_election.cc:288] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [CANDIDATE]: Term 1 election: Requested vote from peers 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65:46387), c233d34625464c8ea28439a829d7721a (127.21.182.66:40383)
I20260810 18:06:50.245496 31063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014b99bdba6f41fcb50c0c7eb9171ec4" candidate_uuid: "38a88b307b7545b5a622c9df9ad2aed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c233d34625464c8ea28439a829d7721a"
I20260810 18:06:50.245487 30908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014b99bdba6f41fcb50c0c7eb9171ec4" candidate_uuid: "38a88b307b7545b5a622c9df9ad2aed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3"
I20260810 18:06:50.245618 31063 raft_consensus.cc:3037] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:50.245618 30908 raft_consensus.cc:3037] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:50.246769 30908 raft_consensus.cc:2454] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38a88b307b7545b5a622c9df9ad2aed5 in term 1.
I20260810 18:06:50.246768 31063 raft_consensus.cc:2454] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38a88b307b7545b5a622c9df9ad2aed5 in term 1.
I20260810 18:06:50.247067 31140 leader_election.cc:302] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a88b307b7545b5a622c9df9ad2aed5, c233d34625464c8ea28439a829d7721a; no voters:
I20260810 18:06:50.247267 31325 raft_consensus.cc:2781] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:50.247360 31325 raft_consensus.cc:686] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 LEADER]: Becoming Leader. State: Replica: 38a88b307b7545b5a622c9df9ad2aed5, State: Running, Role: LEADER
I20260810 18:06:50.247496 31325 consensus_queue.cc:237] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.249490 30735 catalog_manager.cc:5495] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67). New cstate: current_term: 1 leader_uuid: "38a88b307b7545b5a622c9df9ad2aed5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } health_report { overall_health: HEALTHY } } }
Aug 10 18:06:50 dist-test-slave-27sr krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385209, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.126@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
I20260810 18:06:50.295571 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.286029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43152 (local address 127.21.182.126:40091)
0810 18:06:50.286179 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.286184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.286410 (+ 226us) server_negotiation.cc:407] Connection header received
0810 18:06:50.286585 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.286590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.286644 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.286776 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:50.287492 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.288214 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.289136 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.289385 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.289498 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.292133 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:50.292136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:50.292138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:50.292161 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:50.293615 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:50.294194 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:50.294197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:50.294198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:50.294276 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:50.294588 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:50.294590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:50.294591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:50.294771 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:50.294851 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.295215 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.295428 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":61}
I20260810 18:06:50.295576 31347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.285942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40091 (local address 127.0.0.1:43152)
0810 18:06:50.286305 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:50.286321 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:50.286456 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:50.286795 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:50.286798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:50.287272 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:50.287367 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.287374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.288322 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.288328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.288997 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.289007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.289505 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.289508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.289603 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.290269 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:50.290282 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:50.291984 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:50.293734 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:50.293747 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:50.293755 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:50.294089 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:50.294389 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:50.294392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:50.294394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:50.294496 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:50.294884 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:50.294891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:50.295036 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 18:06:50.295172 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 18:06:50.295352 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":3,"thread_start_us":138,"threads_started":1}
I20260810 18:06:50.300333 30735 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:43152:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 18:06:50.301141 30735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:50.306507 30885 tablet_service.cc:1467] Processing CreateTablet for tablet 5fd0b270dcca4791b9891fb57e4ad691 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=adb6afadfd554634a2d9a3bfe1903aa6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:06:50.306507 31040 tablet_service.cc:1467] Processing CreateTablet for tablet 5fd0b270dcca4791b9891fb57e4ad691 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=adb6afadfd554634a2d9a3bfe1903aa6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:06:50.306952 31040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd0b270dcca4791b9891fb57e4ad691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.306952 30885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd0b270dcca4791b9891fb57e4ad691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.308319 31197 tablet_service.cc:1467] Processing CreateTablet for tablet 5fd0b270dcca4791b9891fb57e4ad691 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=adb6afadfd554634a2d9a3bfe1903aa6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:06:50.308698 31197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd0b270dcca4791b9891fb57e4ad691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.310196 31307 tablet_bootstrap.cc:492] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a: Bootstrap starting.
I20260810 18:06:50.310271 31305 tablet_bootstrap.cc:492] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Bootstrap starting.
I20260810 18:06:50.311311 31305 tablet_bootstrap.cc:654] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.311321 31307 tablet_bootstrap.cc:654] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.311868 31304 tablet_bootstrap.cc:492] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5: Bootstrap starting.
I20260810 18:06:50.312853 31304 tablet_bootstrap.cc:654] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.313124 31305 tablet_bootstrap.cc:492] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: No bootstrap required, opened a new log
I20260810 18:06:50.313189 31305 ts_tablet_manager.cc:1397] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 18:06:50.313712 31305 raft_consensus.cc:348] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.313853 31305 raft_consensus.cc:374] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.313896 31305 raft_consensus.cc:729] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c47f0fd74c4923b7a1ba3dd7e41ae3, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.314088 31304 tablet_bootstrap.cc:492] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5: No bootstrap required, opened a new log
I20260810 18:06:50.314041 31305 consensus_queue.cc:260] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.314185 31304 ts_tablet_manager.cc:1397] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 18:06:50.314355 31305 ts_tablet_manager.cc:1428] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:06:50.314587 31309 tablet_replica.cc:439] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } } }
I20260810 18:06:50.314672 31304 raft_consensus.cc:348] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.314750 31309 tablet_replica.cc:442] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:50.314780 31304 raft_consensus.cc:374] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.314815 31304 raft_consensus.cc:729] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a88b307b7545b5a622c9df9ad2aed5, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.314963 31304 consensus_queue.cc:260] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.315201 31316 tablet_replica.cc:439] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } } }
I20260810 18:06:50.315330 31316 tablet_replica.cc:442] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:50.315567 31304 ts_tablet_manager.cc:1428] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:06:50.316608 31307 tablet_bootstrap.cc:492] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a: No bootstrap required, opened a new log
I20260810 18:06:50.316741 31307 ts_tablet_manager.cc:1397] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260810 18:06:50.317278 31307 raft_consensus.cc:348] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.317392 31307 raft_consensus.cc:374] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.317425 31307 raft_consensus.cc:729] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c233d34625464c8ea28439a829d7721a, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.317559 31307 consensus_queue.cc:260] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.317826 31312 tablet_replica.cc:439] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } } }
I20260810 18:06:50.317987 31312 tablet_replica.cc:442] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:50.319149 31307 ts_tablet_manager.cc:1428] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260810 18:06:50.456604 31112 tablet.cc:2369] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:06:50.464406 30957 tablet.cc:2369] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:06:50.473421 31274 tablet.cc:2369] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:06:50.505429 31312 raft_consensus.cc:482] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:50.505616 31312 raft_consensus.cc:504] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.506958 31312 leader_election.cc:288] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65:46387), 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67:33069)
I20260810 18:06:50.511513 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.507386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:48951 (local address 127.21.182.67:33069)
0810 18:06:50.507544 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.507550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.507720 (+ 170us) server_negotiation.cc:407] Connection header received
0810 18:06:50.507936 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.507940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.507993 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.508078 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:50.508399 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.508986 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.510428 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.510930 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.511063 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.511112 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.511310 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.511381 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:06:50.511514 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.507403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:43651 (local address 127.21.182.65:46387)
0810 18:06:50.507562 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.507567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.507926 (+ 359us) server_negotiation.cc:407] Connection header received
0810 18:06:50.508124 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.508127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.508180 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.508277 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:06:50.508557 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.509062 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.510442 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.510899 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.510977 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.511019 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.511305 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.511386 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:06:50.511540 31351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.507248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:33069 (local address 127.21.182.66:48951)
0810 18:06:50.507618 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:50.507631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:50.507781 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:50.508092 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:50.508095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:50.508112 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:50.508257 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.508262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.509095 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.509100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.510322 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.510333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.511059 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.511062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.511159 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.511190 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 18:06:50.511285 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 18:06:50.511380 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":46,"thread_start_us":76,"threads_started":1}
I20260810 18:06:50.511548 31352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.507248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:46387 (local address 127.21.182.66:43651)
0810 18:06:50.507785 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:50.507799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:50.507966 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:50.508289 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:50.508292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:50.508308 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:06:50.508437 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.508442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.509197 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.509200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.510272 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.510282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.511100 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.511102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.511173 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.511191 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:06:50.511281 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 18:06:50.511372 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":254,"thread_start_us":114,"threads_started":1}
I20260810 18:06:50.511996 31218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd0b270dcca4791b9891fb57e4ad691" candidate_uuid: "c233d34625464c8ea28439a829d7721a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a88b307b7545b5a622c9df9ad2aed5" is_pre_election: true
I20260810 18:06:50.512176 31218 raft_consensus.cc:2454] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c233d34625464c8ea28439a829d7721a in term 0.
I20260810 18:06:50.512310 30908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd0b270dcca4791b9891fb57e4ad691" candidate_uuid: "c233d34625464c8ea28439a829d7721a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" is_pre_election: true
I20260810 18:06:50.512457 30908 raft_consensus.cc:2454] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c233d34625464c8ea28439a829d7721a in term 0.
I20260810 18:06:50.512588 30987 leader_election.cc:302] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a88b307b7545b5a622c9df9ad2aed5, c233d34625464c8ea28439a829d7721a; no voters:
I20260810 18:06:50.512874 31312 raft_consensus.cc:2781] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:50.512934 31312 raft_consensus.cc:482] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:50.512962 31312 raft_consensus.cc:3037] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:50.513998 31312 raft_consensus.cc:504] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.514376 31312 leader_election.cc:288] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [CANDIDATE]: Term 1 election: Requested vote from peers 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65:46387), 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67:33069)
I20260810 18:06:50.514734 31218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd0b270dcca4791b9891fb57e4ad691" candidate_uuid: "c233d34625464c8ea28439a829d7721a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a88b307b7545b5a622c9df9ad2aed5"
I20260810 18:06:50.514731 30908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fd0b270dcca4791b9891fb57e4ad691" candidate_uuid: "c233d34625464c8ea28439a829d7721a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3"
I20260810 18:06:50.514854 31218 raft_consensus.cc:3037] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:50.514853 30908 raft_consensus.cc:3037] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:50.516026 30908 raft_consensus.cc:2454] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c233d34625464c8ea28439a829d7721a in term 1.
I20260810 18:06:50.516062 31218 raft_consensus.cc:2454] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c233d34625464c8ea28439a829d7721a in term 1.
I20260810 18:06:50.516341 30987 leader_election.cc:302] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a88b307b7545b5a622c9df9ad2aed5, c233d34625464c8ea28439a829d7721a; no voters:
I20260810 18:06:50.516597 31312 raft_consensus.cc:2781] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:50.516839 31312 raft_consensus.cc:686] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [term 1 LEADER]: Becoming Leader. State: Replica: c233d34625464c8ea28439a829d7721a, State: Running, Role: LEADER
I20260810 18:06:50.517118 31312 consensus_queue.cc:237] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.518424 31353 tablet_replica.cc:439] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a: TxnStatusTablet state changed. Reason: New leader c233d34625464c8ea28439a829d7721a. Latest consensus state: current_term: 1 leader_uuid: "c233d34625464c8ea28439a829d7721a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } } }
I20260810 18:06:50.518570 31353 tablet_replica.cc:442] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:06:50.518972 31355 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:06:50.519147 31355 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 18:06:50.519737 30733 catalog_manager.cc:5495] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a reported cstate change: term changed from 0 to 1, leader changed from <none> to c233d34625464c8ea28439a829d7721a (127.21.182.66). New cstate: current_term: 1 leader_uuid: "c233d34625464c8ea28439a829d7721a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } health_report { overall_health: UNKNOWN } } }
I20260810 18:06:50.568419 31347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.563052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:40383 (local address 127.0.0.1:55420)
0810 18:06:50.563328 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:50.563344 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:50.563453 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:50.563882 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:50.563905 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:50.563916 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:06:50.564038 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.564045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.565211 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.565215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.566212 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.566225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.566912 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.566916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.567017 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.567099 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:06:50.568065 (+ 966us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:06:50.568083 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:06:50.568159 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 18:06:50.568266 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260810 18:06:50.568420 31301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.563262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.21.182.66:40383)
0810 18:06:50.563422 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.563429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.563447 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:06:50.563635 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.563641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.563720 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.563850 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:50.564171 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.565056 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.566387 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.566773 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.566903 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.567301 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:50.567975 (+ 674us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:50.568095 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.568221 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.568275 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:06:50.571933 30908 raft_consensus.cc:1261] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Refusing update from remote peer c233d34625464c8ea28439a829d7721a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:50.572103 31218 raft_consensus.cc:1261] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Refusing update from remote peer c233d34625464c8ea28439a829d7721a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:50.572494 31353 consensus_queue.cc:1035] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:50.574046 31309 tablet_replica.cc:439] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: TxnStatusTablet state changed. Reason: New leader c233d34625464c8ea28439a829d7721a. Latest consensus state: current_term: 1 leader_uuid: "c233d34625464c8ea28439a829d7721a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } } }
I20260810 18:06:50.574173 31309 tablet_replica.cc:442] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:50.575109 31353 consensus_queue.cc:1035] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a [LEADER]: Connected to new peer: Peer: permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:50.576238 31312 tablet_replica.cc:439] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c233d34625464c8ea28439a829d7721a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } } }
I20260810 18:06:50.576357 31312 tablet_replica.cc:442] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:06:50.576440 31309 tablet_replica.cc:439] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c233d34625464c8ea28439a829d7721a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } } }
I20260810 18:06:50.576543 31309 tablet_replica.cc:442] T 5fd0b270dcca4791b9891fb57e4ad691 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:50.577032 31316 tablet_replica.cc:439] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5: TxnStatusTablet state changed. Reason: New leader c233d34625464c8ea28439a829d7721a. Latest consensus state: current_term: 1 leader_uuid: "c233d34625464c8ea28439a829d7721a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } } }
I20260810 18:06:50.577145 31316 tablet_replica.cc:442] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:50.581596 31312 tablet_replica.cc:439] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c233d34625464c8ea28439a829d7721a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } } }
I20260810 18:06:50.581725 31312 tablet_replica.cc:442] T 5fd0b270dcca4791b9891fb57e4ad691 P c233d34625464c8ea28439a829d7721a: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:06:50.583073 31319 mvcc.cc:205] Tried to move back new op lower bound from 7317033822496460800 to 7317033822280724480. Current Snapshot: MvccSnapshot[applied={T|T < 7317033822496460800}]
I20260810 18:06:50.583269 31316 tablet_replica.cc:439] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c233d34625464c8ea28439a829d7721a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } } }
I20260810 18:06:50.583391 31316 tablet_replica.cc:442] T 5fd0b270dcca4791b9891fb57e4ad691 P 38a88b307b7545b5a622c9df9ad2aed5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:06:50.594594 31364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.590866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:40383 (local address 127.21.182.67:34711)
0810 18:06:50.591267 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:50.591281 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:50.591419 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:50.591691 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:50.591694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:50.591702 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:06:50.591824 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.591835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.592643 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.592646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.593290 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.593300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.593795 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.593799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.593885 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.593933 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:06:50.594258 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:06:50.594268 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 18:06:50.594351 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 18:06:50.594445 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":85,"threads_started":1}
I20260810 18:06:50.594579 31301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.591012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:34711 (local address 127.21.182.66:40383)
0810 18:06:50.591130 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.591134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.591359 (+ 225us) server_negotiation.cc:407] Connection header received
0810 18:06:50.591528 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.591531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.591587 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.591665 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:50.592002 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.592517 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.593425 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.593658 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.593778 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.594087 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:50.594169 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:50.594251 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.594377 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.594431 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 18:06:50.602572 30908 raft_consensus.cc:1261] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Refusing update from remote peer 38a88b307b7545b5a622c9df9ad2aed5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:50.602699 31063 raft_consensus.cc:1261] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Refusing update from remote peer 38a88b307b7545b5a622c9df9ad2aed5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:50.603052 31325 consensus_queue.cc:1035] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:50.603210 31316 consensus_queue.cc:1035] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:50.627380 31369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.623003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:33069 (local address 127.21.182.66:47337)
0810 18:06:50.623428 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:50.623442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:50.623574 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:50.623997 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:50.624000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:50.624009 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:06:50.624166 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.624174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.625135 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.625139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.626112 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:50.626124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.626581 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.626584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:50.626687 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.626754 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:06:50.627053 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:06:50.627060 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:06:50.627130 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 18:06:50.627218 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":113,"threads_started":1}
I20260810 18:06:50.627358 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.623114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:47337 (local address 127.21.182.67:33069)
0810 18:06:50.623298 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.623304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.623521 (+ 217us) server_negotiation.cc:407] Connection header received
0810 18:06:50.623697 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.623775 (+ 78us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.623848 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.623986 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:50.624342 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.625003 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.626247 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.626461 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.626546 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.626894 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:50.626963 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:50.627020 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.627192 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.627247 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 18:06:50.650089 30733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:50.650300 30733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:50.652676 30733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923: Sending DeleteTablet for 3 replicas of tablet 014b99bdba6f41fcb50c0c7eb9171ec4
I20260810 18:06:50.653375 30885 tablet_service.cc:1514] Processing DeleteTablet for tablet 014b99bdba6f41fcb50c0c7eb9171ec4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:50 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:60452
I20260810 18:06:50.653577 31331 tablet_replica.cc:330] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: stopping tablet replica
I20260810 18:06:50.653575 31040 tablet_service.cc:1514] Processing DeleteTablet for tablet 014b99bdba6f41fcb50c0c7eb9171ec4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:50 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:55408
I20260810 18:06:50.653698 31331 raft_consensus.cc:2229] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:50.653816 31331 raft_consensus.cc:2258] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:50.653789 31197 tablet_service.cc:1514] Processing DeleteTablet for tablet 014b99bdba6f41fcb50c0c7eb9171ec4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:50 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:46250
I20260810 18:06:50.653925 31329 tablet_replica.cc:330] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a: stopping tablet replica
I20260810 18:06:50.653980 31332 tablet_replica.cc:330] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5: stopping tablet replica
I20260810 18:06:50.654050 31329 raft_consensus.cc:2229] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:50.654107 31332 raft_consensus.cc:2229] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:06:50.654167 31329 raft_consensus.cc:2258] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:50.654330 31332 raft_consensus.cc:2258] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:50.655061 31329 ts_tablet_manager.cc:1905] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:50.655218 31332 ts_tablet_manager.cc:1905] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:50.658186 31329 ts_tablet_manager.cc:1918] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:06:50.658277 31329 log.cc:1196] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/014b99bdba6f41fcb50c0c7eb9171ec4
I20260810 18:06:50.658564 31329 ts_tablet_manager.cc:1939] T 014b99bdba6f41fcb50c0c7eb9171ec4 P c233d34625464c8ea28439a829d7721a: Deleting consensus metadata
I20260810 18:06:50.658735 31332 ts_tablet_manager.cc:1918] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:06:50.658818 31332 log.cc:1196] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/014b99bdba6f41fcb50c0c7eb9171ec4
I20260810 18:06:50.659091 31332 ts_tablet_manager.cc:1939] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 38a88b307b7545b5a622c9df9ad2aed5: Deleting consensus metadata
I20260810 18:06:50.659366 30718 catalog_manager.cc:4841] TS c233d34625464c8ea28439a829d7721a (127.21.182.66:40383): tablet 014b99bdba6f41fcb50c0c7eb9171ec4 (table test-table [id=dac868aa539d41e0aa89524fc86f2877]) successfully deleted
I20260810 18:06:50.660584 31331 ts_tablet_manager.cc:1905] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:50.664115 31331 ts_tablet_manager.cc:1918] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:06:50.664229 31331 log.cc:1196] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/wal/wals/014b99bdba6f41fcb50c0c7eb9171ec4
I20260810 18:06:50.664248 30720 catalog_manager.cc:4841] TS 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67:33069): tablet 014b99bdba6f41fcb50c0c7eb9171ec4 (table test-table [id=dac868aa539d41e0aa89524fc86f2877]) successfully deleted
I20260810 18:06:50.664508 31331 ts_tablet_manager.cc:1939] T 014b99bdba6f41fcb50c0c7eb9171ec4 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Deleting consensus metadata
I20260810 18:06:50.665475 30718 catalog_manager.cc:4841] TS 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65:46387): tablet 014b99bdba6f41fcb50c0c7eb9171ec4 (table test-table [id=dac868aa539d41e0aa89524fc86f2877]) successfully deleted
Aug 10 18:06:50 dist-test-slave-27sr krb5kdc[30685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:06:50 dist-test-slave-27sr krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:50 dist-test-slave-27sr krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:06:50.686085 30787 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:50.678508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.21.182.126:40091)
0810 18:06:50.678675 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.678681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.678993 (+ 312us) server_negotiation.cc:407] Connection header received
0810 18:06:50.679198 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.679208 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.679261 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.679359 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:50.680252 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.680861 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.681868 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.682118 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.682252 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.685943 (+ 3691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43162: BlockingRecv error: recv got EOF from 127.0.0.1:43162 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 18:06:50 dist-test-slave-27sr krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM
I20260810 18:06:50.697466 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.688057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43168 (local address 127.21.182.126:40091)
0810 18:06:50.688193 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.688197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.688386 (+ 189us) server_negotiation.cc:407] Connection header received
0810 18:06:50.688551 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.688554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.688611 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.688707 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:50.689463 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.690031 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.690925 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.691174 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.691291 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.693976 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:50.693980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:50.693982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:50.694009 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:50.695566 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:50.696094 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:50.696098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:50.696102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:50.696193 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:50.696497 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:50.696500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:50.696504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:50.696695 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:50.696916 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.697167 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.697297 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:06:50.700706 30733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:06:50.701202 30733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:50.706310 30885 tablet_service.cc:1467] Processing CreateTablet for tablet 1005baef4d8449159018e1dc71b98043 (DEFAULT_TABLE table=test-table [id=8b2ed7828afe47718900a857036856fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:50.706522 31040 tablet_service.cc:1467] Processing CreateTablet for tablet 1005baef4d8449159018e1dc71b98043 (DEFAULT_TABLE table=test-table [id=8b2ed7828afe47718900a857036856fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:50.706669 30885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1005baef4d8449159018e1dc71b98043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.706801 31040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1005baef4d8449159018e1dc71b98043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.709390 31307 tablet_bootstrap.cc:492] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a: Bootstrap starting.
I20260810 18:06:50.709592 31197 tablet_service.cc:1467] Processing CreateTablet for tablet 1005baef4d8449159018e1dc71b98043 (DEFAULT_TABLE table=test-table [id=8b2ed7828afe47718900a857036856fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:50.709875 31197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1005baef4d8449159018e1dc71b98043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.710291 31307 tablet_bootstrap.cc:654] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.712319 31305 tablet_bootstrap.cc:492] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Bootstrap starting.
I20260810 18:06:50.712482 31307 tablet_bootstrap.cc:492] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a: No bootstrap required, opened a new log
I20260810 18:06:50.712581 31307 ts_tablet_manager.cc:1397] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 18:06:50.712862 31304 tablet_bootstrap.cc:492] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5: Bootstrap starting.
I20260810 18:06:50.713009 31307 raft_consensus.cc:348] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.713135 31307 raft_consensus.cc:374] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.713176 31307 raft_consensus.cc:729] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c233d34625464c8ea28439a829d7721a, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.713297 31305 tablet_bootstrap.cc:654] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.713338 31307 consensus_queue.cc:260] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.713631 31307 ts_tablet_manager.cc:1428] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:06:50.713850 31304 tablet_bootstrap.cc:654] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.715078 31305 tablet_bootstrap.cc:492] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: No bootstrap required, opened a new log
I20260810 18:06:50.715179 31305 ts_tablet_manager.cc:1397] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 18:06:50.715670 31305 raft_consensus.cc:348] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.715808 31305 raft_consensus.cc:374] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.715857 31305 raft_consensus.cc:729] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c47f0fd74c4923b7a1ba3dd7e41ae3, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.716039 31305 consensus_queue.cc:260] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.716187 31304 tablet_bootstrap.cc:492] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5: No bootstrap required, opened a new log
I20260810 18:06:50.716290 31304 ts_tablet_manager.cc:1397] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 18:06:50.716450 31305 ts_tablet_manager.cc:1428] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260810 18:06:50.716682 31304 raft_consensus.cc:348] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.716827 31304 raft_consensus.cc:374] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.716861 31304 raft_consensus.cc:729] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a88b307b7545b5a622c9df9ad2aed5, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.716970 31304 consensus_queue.cc:260] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.717223 31304 ts_tablet_manager.cc:1428] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:06:50.867010 31309 raft_consensus.cc:482] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:50.867237 31309 raft_consensus.cc:504] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.868001 31309 leader_election.cc:288] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c233d34625464c8ea28439a829d7721a (127.21.182.66:40383), 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67:33069)
I20260810 18:06:50.868474 31218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1005baef4d8449159018e1dc71b98043" candidate_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a88b307b7545b5a622c9df9ad2aed5" is_pre_election: true
I20260810 18:06:50.868459 31063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1005baef4d8449159018e1dc71b98043" candidate_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c233d34625464c8ea28439a829d7721a" is_pre_election: true
I20260810 18:06:50.868628 31063 raft_consensus.cc:2454] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c47f0fd74c4923b7a1ba3dd7e41ae3 in term 0.
I20260810 18:06:50.868628 31218 raft_consensus.cc:2454] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c47f0fd74c4923b7a1ba3dd7e41ae3 in term 0.
I20260810 18:06:50.868985 30833 leader_election.cc:302] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c47f0fd74c4923b7a1ba3dd7e41ae3, c233d34625464c8ea28439a829d7721a; no voters:
I20260810 18:06:50.869210 31309 raft_consensus.cc:2781] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:50.869290 31309 raft_consensus.cc:482] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:50.869349 31309 raft_consensus.cc:3037] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:50.870612 31309 raft_consensus.cc:504] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.871017 31309 leader_election.cc:288] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [CANDIDATE]: Term 1 election: Requested vote from peers c233d34625464c8ea28439a829d7721a (127.21.182.66:40383), 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67:33069)
I20260810 18:06:50.871461 31063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1005baef4d8449159018e1dc71b98043" candidate_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c233d34625464c8ea28439a829d7721a"
I20260810 18:06:50.871466 31218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1005baef4d8449159018e1dc71b98043" candidate_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a88b307b7545b5a622c9df9ad2aed5"
I20260810 18:06:50.871624 31218 raft_consensus.cc:3037] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:50.871624 31063 raft_consensus.cc:3037] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:50.872970 31218 raft_consensus.cc:2454] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c47f0fd74c4923b7a1ba3dd7e41ae3 in term 1.
I20260810 18:06:50.872970 31063 raft_consensus.cc:2454] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c47f0fd74c4923b7a1ba3dd7e41ae3 in term 1.
I20260810 18:06:50.873307 30833 leader_election.cc:302] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c47f0fd74c4923b7a1ba3dd7e41ae3, c233d34625464c8ea28439a829d7721a; no voters:
I20260810 18:06:50.873567 31309 raft_consensus.cc:2781] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:50.873785 31309 raft_consensus.cc:686] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 LEADER]: Becoming Leader. State: Replica: 02c47f0fd74c4923b7a1ba3dd7e41ae3, State: Running, Role: LEADER
I20260810 18:06:50.874073 31309 consensus_queue.cc:237] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.876603 30735 catalog_manager.cc:5495] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65). New cstate: current_term: 1 leader_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } health_report { overall_health: UNKNOWN } } }
I20260810 18:06:50.906692 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:50.902829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.21.182.65:46387)
0810 18:06:50.902967 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:50.902976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:50.902992 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:06:50.903085 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:50.903087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:50.903173 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:50.903283 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:50.903550 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.904167 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.905088 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:50.905341 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:50.905431 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:50.905723 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:50.906321 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:50.906418 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:50.906489 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 18:06:50.906549 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:06:50.909795 31063 raft_consensus.cc:1261] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Refusing update from remote peer 02c47f0fd74c4923b7a1ba3dd7e41ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:50.909874 31218 raft_consensus.cc:1261] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Refusing update from remote peer 02c47f0fd74c4923b7a1ba3dd7e41ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:50.910394 31390 consensus_queue.cc:1035] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:50.910650 31309 consensus_queue.cc:1035] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:50.924515 30733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:50.924698 30733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:50.927266 30733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923: Sending DeleteTablet for 3 replicas of tablet 1005baef4d8449159018e1dc71b98043
I20260810 18:06:50.928184 31040 tablet_service.cc:1514] Processing DeleteTablet for tablet 1005baef4d8449159018e1dc71b98043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:50 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:55408
I20260810 18:06:50.928309 30885 tablet_service.cc:1514] Processing DeleteTablet for tablet 1005baef4d8449159018e1dc71b98043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:50 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:60452
I20260810 18:06:50.928447 31329 tablet_replica.cc:330] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a: stopping tablet replica
I20260810 18:06:50.928516 31331 tablet_replica.cc:330] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: stopping tablet replica
I20260810 18:06:50.928584 31329 raft_consensus.cc:2229] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:50.928632 31331 raft_consensus.cc:2229] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:06:50.928754 31329 raft_consensus.cc:2258] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:50.928817 31197 tablet_service.cc:1514] Processing DeleteTablet for tablet 1005baef4d8449159018e1dc71b98043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:50 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:46250
I20260810 18:06:50.928953 31331 raft_consensus.cc:2258] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:50.929067 31332 tablet_replica.cc:330] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5: stopping tablet replica
I20260810 18:06:50.929200 31332 raft_consensus.cc:2229] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:50.929363 31332 raft_consensus.cc:2258] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:50.929745 30733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 18:06:50.930143 31331 ts_tablet_manager.cc:1905] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:50.930473 31329 ts_tablet_manager.cc:1905] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:50.930714 31332 ts_tablet_manager.cc:1905] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:50.933537 31329 ts_tablet_manager.cc:1918] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:50.933648 31329 log.cc:1196] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/1005baef4d8449159018e1dc71b98043
I20260810 18:06:50.933866 31332 ts_tablet_manager.cc:1918] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:50.933966 31332 log.cc:1196] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/1005baef4d8449159018e1dc71b98043
I20260810 18:06:50.933955 31329 ts_tablet_manager.cc:1939] T 1005baef4d8449159018e1dc71b98043 P c233d34625464c8ea28439a829d7721a: Deleting consensus metadata
I20260810 18:06:50.934486 31332 ts_tablet_manager.cc:1939] T 1005baef4d8449159018e1dc71b98043 P 38a88b307b7545b5a622c9df9ad2aed5: Deleting consensus metadata
W20260810 18:06:50.935192 30733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:06:50.935417 30720 catalog_manager.cc:4841] TS 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67:33069): tablet 1005baef4d8449159018e1dc71b98043 (table test-table [id=8b2ed7828afe47718900a857036856fa]) successfully deleted
I20260810 18:06:50.940163 30718 catalog_manager.cc:4841] TS c233d34625464c8ea28439a829d7721a (127.21.182.66:40383): tablet 1005baef4d8449159018e1dc71b98043 (table test-table [id=8b2ed7828afe47718900a857036856fa]) successfully deleted
I20260810 18:06:50.942723 31331 ts_tablet_manager.cc:1918] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:06:50.942824 31331 log.cc:1196] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-0/wal/wals/1005baef4d8449159018e1dc71b98043
I20260810 18:06:50.943111 31331 ts_tablet_manager.cc:1939] T 1005baef4d8449159018e1dc71b98043 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Deleting consensus metadata
I20260810 18:06:50.944373 31197 tablet_service.cc:1467] Processing CreateTablet for tablet 8af606926f054971b3c1be438faf5968 (DEFAULT_TABLE table=test-table [id=c37d91c917d84a8b99a05a422514161b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:50.944727 31197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af606926f054971b3c1be438faf5968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.944777 30718 catalog_manager.cc:4841] TS 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65:46387): tablet 1005baef4d8449159018e1dc71b98043 (table test-table [id=8b2ed7828afe47718900a857036856fa]) successfully deleted
I20260810 18:06:50.947705 30885 tablet_service.cc:1467] Processing CreateTablet for tablet 8af606926f054971b3c1be438faf5968 (DEFAULT_TABLE table=test-table [id=c37d91c917d84a8b99a05a422514161b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:50.948027 30885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af606926f054971b3c1be438faf5968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.948143 31040 tablet_service.cc:1467] Processing CreateTablet for tablet 8af606926f054971b3c1be438faf5968 (DEFAULT_TABLE table=test-table [id=c37d91c917d84a8b99a05a422514161b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:06:50.948460 31040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af606926f054971b3c1be438faf5968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:50.950599 31305 tablet_bootstrap.cc:492] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Bootstrap starting.
I20260810 18:06:50.950951 31307 tablet_bootstrap.cc:492] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a: Bootstrap starting.
I20260810 18:06:50.951690 31305 tablet_bootstrap.cc:654] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.951822 31307 tablet_bootstrap.cc:654] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.952975 31304 tablet_bootstrap.cc:492] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5: Bootstrap starting.
I20260810 18:06:50.953289 31307 tablet_bootstrap.cc:492] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a: No bootstrap required, opened a new log
I20260810 18:06:50.953395 31307 ts_tablet_manager.cc:1397] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:06:50.953873 31307 raft_consensus.cc:348] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.953984 31304 tablet_bootstrap.cc:654] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:50.954010 31307 raft_consensus.cc:374] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.954056 31307 raft_consensus.cc:729] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c233d34625464c8ea28439a829d7721a, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.954187 31307 consensus_queue.cc:260] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.954501 31307 ts_tablet_manager.cc:1428] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:06:50.956359 31305 tablet_bootstrap.cc:492] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: No bootstrap required, opened a new log
I20260810 18:06:50.956499 31305 ts_tablet_manager.cc:1397] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260810 18:06:50.957043 31305 raft_consensus.cc:348] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.957180 31305 raft_consensus.cc:374] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.957233 31305 raft_consensus.cc:729] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c47f0fd74c4923b7a1ba3dd7e41ae3, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.957468 31304 tablet_bootstrap.cc:492] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5: No bootstrap required, opened a new log
I20260810 18:06:50.957564 31304 ts_tablet_manager.cc:1397] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260810 18:06:50.958015 31304 raft_consensus.cc:348] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.958148 31304 raft_consensus.cc:374] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:50.958189 31304 raft_consensus.cc:729] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a88b307b7545b5a622c9df9ad2aed5, State: Initialized, Role: FOLLOWER
I20260810 18:06:50.958233 31305 consensus_queue.cc:260] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.958389 31304 consensus_queue.cc:260] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:50.959208 31305 ts_tablet_manager.cc:1428] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260810 18:06:50.959524 31304 ts_tablet_manager.cc:1428] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:06:51.233671 31309 raft_consensus.cc:482] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:51.233848 31353 raft_consensus.cc:482] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:06:51.233908 31309 raft_consensus.cc:504] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:51.234071 31353 raft_consensus.cc:504] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:51.234530 31309 leader_election.cc:288] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c233d34625464c8ea28439a829d7721a (127.21.182.66:40383), 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67:33069)
I20260810 18:06:51.234583 31353 leader_election.cc:288] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65:46387), 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67:33069)
I20260810 18:06:51.235098 31218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af606926f054971b3c1be438faf5968" candidate_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a88b307b7545b5a622c9df9ad2aed5" is_pre_election: true
I20260810 18:06:51.235096 30908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af606926f054971b3c1be438faf5968" candidate_uuid: "c233d34625464c8ea28439a829d7721a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" is_pre_election: true
I20260810 18:06:51.235241 30908 raft_consensus.cc:2454] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c233d34625464c8ea28439a829d7721a in term 0.
I20260810 18:06:51.235224 31063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af606926f054971b3c1be438faf5968" candidate_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c233d34625464c8ea28439a829d7721a" is_pre_election: true
I20260810 18:06:51.235221 31217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af606926f054971b3c1be438faf5968" candidate_uuid: "c233d34625464c8ea28439a829d7721a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a88b307b7545b5a622c9df9ad2aed5" is_pre_election: true
I20260810 18:06:51.235364 31063 raft_consensus.cc:2454] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c47f0fd74c4923b7a1ba3dd7e41ae3 in term 0.
I20260810 18:06:51.235242 31218 raft_consensus.cc:2454] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c47f0fd74c4923b7a1ba3dd7e41ae3 in term 0.
I20260810 18:06:51.235589 30985 leader_election.cc:302] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c47f0fd74c4923b7a1ba3dd7e41ae3, c233d34625464c8ea28439a829d7721a; no voters:
I20260810 18:06:51.235704 30833 leader_election.cc:302] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c47f0fd74c4923b7a1ba3dd7e41ae3, c233d34625464c8ea28439a829d7721a; no voters:
I20260810 18:06:51.235883 31309 raft_consensus.cc:2781] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:51.235967 31309 raft_consensus.cc:482] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:51.236023 31309 raft_consensus.cc:3037] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:51.236068 31353 raft_consensus.cc:2781] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:06:51.236138 31353 raft_consensus.cc:482] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:06:51.236187 31353 raft_consensus.cc:3037] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:51.237112 31309 raft_consensus.cc:504] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:51.237212 31353 raft_consensus.cc:504] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:51.237565 31309 leader_election.cc:288] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [CANDIDATE]: Term 1 election: Requested vote from peers c233d34625464c8ea28439a829d7721a (127.21.182.66:40383), 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67:33069)
I20260810 18:06:51.237673 31353 leader_election.cc:288] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [CANDIDATE]: Term 1 election: Requested vote from peers 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65:46387), 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67:33069)
I20260810 18:06:51.237910 31063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af606926f054971b3c1be438faf5968" candidate_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c233d34625464c8ea28439a829d7721a"
I20260810 18:06:51.237982 31218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af606926f054971b3c1be438faf5968" candidate_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a88b307b7545b5a622c9df9ad2aed5"
I20260810 18:06:51.238042 30908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af606926f054971b3c1be438faf5968" candidate_uuid: "c233d34625464c8ea28439a829d7721a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3"
I20260810 18:06:51.238085 31063 raft_consensus.cc:2379] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 02c47f0fd74c4923b7a1ba3dd7e41ae3 in current term 1: Already voted for candidate c233d34625464c8ea28439a829d7721a in this term.
I20260810 18:06:51.238098 31217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af606926f054971b3c1be438faf5968" candidate_uuid: "c233d34625464c8ea28439a829d7721a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a88b307b7545b5a622c9df9ad2aed5"
I20260810 18:06:51.238215 30908 raft_consensus.cc:2379] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c233d34625464c8ea28439a829d7721a in current term 1: Already voted for candidate 02c47f0fd74c4923b7a1ba3dd7e41ae3 in this term.
I20260810 18:06:51.238107 31218 raft_consensus.cc:3037] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:51.238765 30987 leader_election.cc:302] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c233d34625464c8ea28439a829d7721a; no voters: 02c47f0fd74c4923b7a1ba3dd7e41ae3, 38a88b307b7545b5a622c9df9ad2aed5
I20260810 18:06:51.238984 31353 raft_consensus.cc:2726] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 18:06:51.239740 31218 raft_consensus.cc:2454] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c47f0fd74c4923b7a1ba3dd7e41ae3 in term 1.
I20260810 18:06:51.240103 30835 leader_election.cc:302] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02c47f0fd74c4923b7a1ba3dd7e41ae3, 38a88b307b7545b5a622c9df9ad2aed5; no voters: c233d34625464c8ea28439a829d7721a
I20260810 18:06:51.240368 31309 raft_consensus.cc:2781] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:51.240483 31309 raft_consensus.cc:686] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 LEADER]: Becoming Leader. State: Replica: 02c47f0fd74c4923b7a1ba3dd7e41ae3, State: Running, Role: LEADER
I20260810 18:06:51.240640 31309 consensus_queue.cc:237] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } }
I20260810 18:06:51.242630 30734 catalog_manager.cc:5495] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65). New cstate: current_term: 1 leader_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c47f0fd74c4923b7a1ba3dd7e41ae3" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 46387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 } health_report { overall_health: UNKNOWN } } }
I20260810 18:06:51.271154 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:51.267051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:40383 (local address 127.21.182.65:35647)
0810 18:06:51.267550 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:51.267567 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:51.267747 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:51.268154 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:51.268156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:51.268164 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:06:51.268270 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:51.268275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:51.269172 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:51.269175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:51.269846 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:51.269856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:51.270299 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:51.270302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:51.270367 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:51.270418 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:06:51.270881 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:06:51.270887 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:06:51.270940 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 18:06:51.271016 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":203,"threads_started":1}
I20260810 18:06:51.271181 31405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:51.267186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:35647 (local address 127.21.182.66:40383)
0810 18:06:51.267533 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:51.267537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:51.267780 (+ 243us) server_negotiation.cc:407] Connection header received
0810 18:06:51.267983 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:51.267986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:51.268042 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:51.268126 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:51.268386 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:51.269064 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:51.269965 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:51.270191 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:51.270280 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:51.270596 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:51.270764 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:51.270842 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:51.270987 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 18:06:51.271045 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":88,"threads_started":1}
I20260810 18:06:51.277956 31218 raft_consensus.cc:1261] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Refusing update from remote peer 02c47f0fd74c4923b7a1ba3dd7e41ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:51.278402 31063 raft_consensus.cc:1261] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Refusing update from remote peer 02c47f0fd74c4923b7a1ba3dd7e41ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:06:51.278546 31390 consensus_queue.cc:1035] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38a88b307b7545b5a622c9df9ad2aed5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:51.278901 31309 consensus_queue.cc:1035] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c233d34625464c8ea28439a829d7721a" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:06:51.299520 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:51.295598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:35427 (local address 127.21.182.65:46387)
0810 18:06:51.295861 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:51.295864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:51.296213 (+ 349us) server_negotiation.cc:407] Connection header received
0810 18:06:51.296340 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:51.296343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:51.296389 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:51.296446 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:06:51.296642 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:51.297129 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:51.298277 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:51.298615 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:51.298731 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:51.299039 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:06:51.299123 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:06:51.299196 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:51.299298 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 18:06:51.299360 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 18:06:51.299542 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:51.295496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:46387 (local address 127.21.182.66:35427)
0810 18:06:51.296095 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:51.296113 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:51.296231 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:51.296473 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:51.296475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:51.296481 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:06:51.296576 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:51.296580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:51.297247 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:51.297250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:51.298131 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:51.298145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:51.298817 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:51.298820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:51.298888 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:51.298918 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:06:51.299222 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:06:51.299229 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:06:51.299286 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 18:06:51.299364 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":149,"threads_started":1}
I20260810 18:06:51.323400 30733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:51.323566 30733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:06:51.325826 30733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b8ae763e7a54263948879b57a7d2923: Sending DeleteTablet for 3 replicas of tablet 8af606926f054971b3c1be438faf5968
I20260810 18:06:51.326467 30885 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af606926f054971b3c1be438faf5968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:51 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:60452
I20260810 18:06:51.326525 31040 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af606926f054971b3c1be438faf5968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:51 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:55408
I20260810 18:06:51.326694 31331 tablet_replica.cc:330] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3: stopping tablet replica
I20260810 18:06:51.326722 31329 tablet_replica.cc:330] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a: stopping tablet replica
I20260810 18:06:51.326792 31197 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af606926f054971b3c1be438faf5968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:06:51 UTC) from {username='oryx', principal='oryx/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:46250
I20260810 18:06:51.326843 31331 raft_consensus.cc:2229] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:06:51.326864 31329 raft_consensus.cc:2229] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:51.326988 31329 raft_consensus.cc:2258] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:51.327003 31332 tablet_replica.cc:330] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5: stopping tablet replica
I20260810 18:06:51.327080 31331 raft_consensus.cc:2258] T 8af606926f054971b3c1be438faf5968 P 02c47f0fd74c4923b7a1ba3dd7e41ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:51.327121 31332 raft_consensus.cc:2229] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:06:51.327245 31332 raft_consensus.cc:2258] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:06:51.327445 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 30813
I20260810 18:06:51.331004 31332 ts_tablet_manager.cc:1905] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:51.333649 31329 ts_tablet_manager.cc:1905] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:06:51.334437 31332 ts_tablet_manager.cc:1918] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:06:51.334509 31332 log.cc:1196] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/8af606926f054971b3c1be438faf5968
I20260810 18:06:51.334749 31332 ts_tablet_manager.cc:1939] T 8af606926f054971b3c1be438faf5968 P 38a88b307b7545b5a622c9df9ad2aed5: Deleting consensus metadata
I20260810 18:06:51.335716 30720 catalog_manager.cc:4841] TS 38a88b307b7545b5a622c9df9ad2aed5 (127.21.182.67:33069): tablet 8af606926f054971b3c1be438faf5968 (table test-table [id=c37d91c917d84a8b99a05a422514161b]) successfully deleted
I20260810 18:06:51.336009 31329 ts_tablet_manager.cc:1918] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:06:51.336120 31329 log.cc:1196] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/8af606926f054971b3c1be438faf5968
I20260810 18:06:51.336372 31329 ts_tablet_manager.cc:1939] T 8af606926f054971b3c1be438faf5968 P c233d34625464c8ea28439a829d7721a: Deleting consensus metadata
I20260810 18:06:51.337376 30718 catalog_manager.cc:4841] TS c233d34625464c8ea28439a829d7721a (127.21.182.66:40383): tablet 8af606926f054971b3c1be438faf5968 (table test-table [id=c37d91c917d84a8b99a05a422514161b]) successfully deleted
W20260810 18:06:51.345113 30718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:46387 (error 108)
I20260810 18:06:51.345275 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 30967
I20260810 18:06:51.349160 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:51.348280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:46387 (local address 127.0.0.1:60472)
0810 18:06:51.348906 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:51.348998 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":339,"threads_started":1}
W20260810 18:06:51.349292 30718 catalog_manager.cc:4584] TS 02c47f0fd74c4923b7a1ba3dd7e41ae3 (127.21.182.65:46387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8af606926f054971b3c1be438faf5968: Network error: Client connection negotiation failed: client connection to 127.21.182.65:46387: connect: Connection refused (error 111)
I20260810 18:06:51.358842 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 31120
I20260810 18:06:51.371740 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 30698
2026-08-10T18:06:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3693 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:06:51 dist-test-slave-27sr krb5kdc[31421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:06:51 dist-test-slave-27sr krb5kdc[31421](info): set up 2 sockets
Aug 10 18:06:51 dist-test-slave-27sr krb5kdc[31421](info): commencing operation
krb5kdc: starting...
W20260810 18:06:52.593170 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:06:52 dist-test-slave-27sr krb5kdc[31421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:06:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:06:52.740499 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:36039
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:36039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:52.869863 31434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:52.870215 31434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:52.870298 31434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:52.874975 31434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:52.875077 31434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:52.875118 31434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:52.875155 31434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:52.875192 31434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:52.882699 31434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:36039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:36039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:52.884017 31434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:52.885149 31434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:52.895521 31445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:52.895557 31441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:52.895715 31434 server_base.cc:1034] running on GCE node
W20260810 18:06:52.895938 31440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:52.896633 31434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:52.898082 31434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:52.899290 31434 hybrid_clock.cc:648] HybridClock initialized: now 1786385212899273 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T18:06:52Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1534 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:06:52 dist-test-slave-27sr krb5kdc[31450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:06:52 dist-test-slave-27sr krb5kdc[31450](info): set up 2 sockets
Aug 10 18:06:52 dist-test-slave-27sr krb5kdc[31450](info): commencing operation
krb5kdc: starting...
W20260810 18:06:54.051327 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.009s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:06:54 dist-test-slave-27sr krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:06:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:06:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:06:54.189270 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:35241
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:35241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:54.314913 31466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:54.315313 31466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:54.315397 31466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:54.319816 31466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:54.319967 31466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:54.320041 31466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:54.320099 31466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:54.320140 31466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:54.326088 31466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:35241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:35241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:54.327421 31466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:54.328572 31466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:54.338353 31474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:54.338346 31472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:54.338346 31471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:54.338617 31466 server_base.cc:1034] running on GCE node
I20260810 18:06:54.339476 31466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:54.340876 31466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:54.342100 31466 hybrid_clock.cc:648] HybridClock initialized: now 1786385214342065 us; error 63 us; skew 500 ppm
Aug 10 18:06:54 dist-test-slave-27sr krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385214, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:54.351485 31466 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:06:54.352814 31466 webserver.cc:460] Webserver started at http://127.21.182.126:34729/ using document root <none> and password file <none>
I20260810 18:06:54.353514 31466 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:54.353617 31466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:54.353891 31466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:54.356433 31466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "dfcbf886c36942f59767ed1d35e88eaa"
format_stamp: "Formatted at 2026-08-10 18:06:54 on dist-test-slave-27sr"
server_key: "017876926771c8cf6ae3e3420c0097a9"
server_key_iv: "a9dd805ed3543dee847d4e64cbbaa269"
server_key_version: "encryptionkey@0"
I20260810 18:06:54.357156 31466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "dfcbf886c36942f59767ed1d35e88eaa"
format_stamp: "Formatted at 2026-08-10 18:06:54 on dist-test-slave-27sr"
server_key: "017876926771c8cf6ae3e3420c0097a9"
server_key_iv: "a9dd805ed3543dee847d4e64cbbaa269"
server_key_version: "encryptionkey@0"
I20260810 18:06:54.361761 31466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:06:54.364723 31481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:54.366389 31466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:54.366582 31466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "dfcbf886c36942f59767ed1d35e88eaa"
format_stamp: "Formatted at 2026-08-10 18:06:54 on dist-test-slave-27sr"
server_key: "017876926771c8cf6ae3e3420c0097a9"
server_key_iv: "a9dd805ed3543dee847d4e64cbbaa269"
server_key_version: "encryptionkey@0"
I20260810 18:06:54.366744 31466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:54.388013 31466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:54.391345 31466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:54.391564 31466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:54.399587 31466 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:35241
I20260810 18:06:54.399603 31540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:35241 every 8 connection(s)
I20260810 18:06:54.400678 31466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:54.403679 31541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:06:54.408329 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 31466
I20260810 18:06:54.408452 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:06:54.408762 22233 external_mini_cluster.cc:1428] Setting key 2b525cb84d5be2e540c9c968262abd83
I20260810 18:06:54.410110 31541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa: Bootstrap starting.
I20260810 18:06:54.412761 31541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa: Neither blocks nor log segments found. Creating new log.
I20260810 18:06:54.413592 31541 log.cc:824] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:54.415722 31541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa: No bootstrap required, opened a new log
Aug 10 18:06:54 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:54.420192 31541 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbf886c36942f59767ed1d35e88eaa" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35241 } }
I20260810 18:06:54.420521 31541 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:06:54.420591 31541 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcbf886c36942f59767ed1d35e88eaa, State: Initialized, Role: FOLLOWER
I20260810 18:06:54.421159 31541 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbf886c36942f59767ed1d35e88eaa" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35241 } }
I20260810 18:06:54.421311 31541 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:54.421401 31541 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:54.421510 31541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:06:54.422864 31541 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbf886c36942f59767ed1d35e88eaa" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35241 } }
I20260810 18:06:54.423262 31541 leader_election.cc:302] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfcbf886c36942f59767ed1d35e88eaa; no voters:
I20260810 18:06:54.423671 31541 leader_election.cc:288] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:06:54.423851 31549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:06:54.424158 31549 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 1 LEADER]: Becoming Leader. State: Replica: dfcbf886c36942f59767ed1d35e88eaa, State: Running, Role: LEADER
I20260810 18:06:54.424615 31549 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbf886c36942f59767ed1d35e88eaa" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35241 } }
I20260810 18:06:54.425135 31541 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:54.425616 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:54.410370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53024 (local address 127.21.182.126:35241)
0810 18:06:54.410920 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:54.410938 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:54.410976 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:06:54.411668 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:54.411694 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:54.412147 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:54.412528 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:54.413402 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:54.414460 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:54.415357 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:54.415887 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:54.416093 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:54.419374 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:54.419380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:54.419397 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:54.419431 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:54.421599 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:54.422221 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:54.422228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:54.422233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:54.422379 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:54.422853 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:54.422856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:54.422858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:54.423491 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:54.423711 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:54.424148 (+ 437us) server_negotiation.cc:299] Negotiation successful
0810 18:06:54.424526 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":145,"threads_started":1}
I20260810 18:06:54.426155 31549 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfcbf886c36942f59767ed1d35e88eaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbf886c36942f59767ed1d35e88eaa" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35241 } } }
I20260810 18:06:54.426311 31549 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:54.426522 31549 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfcbf886c36942f59767ed1d35e88eaa. Latest consensus state: current_term: 1 leader_uuid: "dfcbf886c36942f59767ed1d35e88eaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbf886c36942f59767ed1d35e88eaa" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35241 } } }
I20260810 18:06:54.426604 31549 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:54.426808 31559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:54.430428 31559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:54.437613 31559 catalog_manager.cc:1347] Generated new cluster ID: 3b7f3fae424d4700bc6a3d193e66811c
I20260810 18:06:54.437714 31559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:54.448151 31559 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:06:54.449215 31559 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:54.458709 31559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa: Generated new TSK 0
I20260810 18:06:54.459535 31559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:06:54.493613 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35241
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:54.624087 31576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:54.624497 31576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:54.624605 31576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:54.629869 31576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:54.629995 31576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:54.630108 31576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:54.636620 31576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:54.638010 31576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:54.639250 31576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:54.651296 31583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:54.651288 31581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:54.651288 31585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:54.651669 31576 server_base.cc:1034] running on GCE node
I20260810 18:06:54.652200 31576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:54.652985 31576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:54.654160 31576 hybrid_clock.cc:648] HybridClock initialized: now 1786385214654121 us; error 57 us; skew 500 ppm
Aug 10 18:06:54 dist-test-slave-27sr krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385214, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:54.657244 31576 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:06:54.658257 31576 webserver.cc:460] Webserver started at http://127.21.182.65:40251/ using document root <none> and password file <none>
I20260810 18:06:54.659077 31576 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:54.659168 31576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:54.659371 31576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:54.661955 31576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "bc763494852f4638bbf48d9dc7a19f88"
format_stamp: "Formatted at 2026-08-10 18:06:54 on dist-test-slave-27sr"
server_key: "55b1f10542ce3da70d609500652a74bf"
server_key_iv: "51f07ab6b8170c7d0cdca0204ba4d8ea"
server_key_version: "encryptionkey@0"
I20260810 18:06:54.662575 31576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "bc763494852f4638bbf48d9dc7a19f88"
format_stamp: "Formatted at 2026-08-10 18:06:54 on dist-test-slave-27sr"
server_key: "55b1f10542ce3da70d609500652a74bf"
server_key_iv: "51f07ab6b8170c7d0cdca0204ba4d8ea"
server_key_version: "encryptionkey@0"
I20260810 18:06:54.667076 31576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:06:54.669950 31592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:54.671653 31576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:54.671829 31576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "bc763494852f4638bbf48d9dc7a19f88"
format_stamp: "Formatted at 2026-08-10 18:06:54 on dist-test-slave-27sr"
server_key: "55b1f10542ce3da70d609500652a74bf"
server_key_iv: "51f07ab6b8170c7d0cdca0204ba4d8ea"
server_key_version: "encryptionkey@0"
I20260810 18:06:54.672035 31576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:54.686240 31576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:54.689970 31576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:54.690200 31576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:54.690869 31576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:54.691865 31576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:54.691946 31576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:54.692014 31576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:54.692045 31576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:54.701862 31576 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:45849
I20260810 18:06:54.701884 31720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:45849 every 8 connection(s)
I20260810 18:06:54.702912 31576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:54.712885 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 31576
I20260810 18:06:54.713006 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
Aug 10 18:06:54 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385214, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:54.713357 22233 external_mini_cluster.cc:1428] Setting key 7f9bdb2f68e4178d274abf2a4f005e95
I20260810 18:06:54.718081 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:54.705009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:34247 (local address 127.21.182.126:35241)
0810 18:06:54.705151 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:54.705154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:54.706337 (+ 1183us) server_negotiation.cc:407] Connection header received
0810 18:06:54.707301 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:54.707304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:54.707398 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:54.707505 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:54.708827 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:54.709380 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:54.710555 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:54.710803 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:54.710910 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:54.713818 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:54.713822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:54.713824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:54.713854 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:54.715654 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:54.716279 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:54.716285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:54.716290 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:54.716394 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:54.716756 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:54.716762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:54.716767 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:54.717038 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:54.717152 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:54.717759 (+ 607us) server_negotiation.cc:299] Negotiation successful
0810 18:06:54.717905 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:06:54.719038 31724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:54.705404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:34247)
0810 18:06:54.706171 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:54.706216 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:54.707077 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:54.707694 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:54.707702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:54.708155 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:54.708669 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:54.708683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:54.709516 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:54.709519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:54.710423 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:54.710432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:54.710960 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:54.710964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:54.711134 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:54.711713 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:54.711820 (+ 107us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:54.713605 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:54.715801 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:54.715807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:54.715821 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:54.716174 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:54.716497 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:54.716499 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:54.716500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:54.716642 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:54.717168 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:54.717174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:54.717451 (+ 277us) client_negotiation.cc:769] Sending connection context
0810 18:06:54.717698 (+ 247us) client_negotiation.cc:240] Negotiation successful
0810 18:06:54.718080 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":135,"threads_started":1}
I20260810 18:06:54.720495 31721 heartbeater.cc:347] Connected to a master server at 127.21.182.126:35241
I20260810 18:06:54.720763 31721 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:54.721315 31721 heartbeater.cc:511] Master 127.21.182.126:35241 requested a full tablet report, sending...
I20260810 18:06:54.723127 31499 ts_manager.cc:194] Registered new tserver with Master: bc763494852f4638bbf48d9dc7a19f88 (127.21.182.65:45849)
I20260810 18:06:54.724437 31499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:34247
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:06:54.749517 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35241
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:54.874045 31727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:54.874369 31727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:54.874418 31727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:54.878917 31727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:54.879009 31727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:54.879094 31727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:54.886319 31727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:54.887590 31727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:54.888931 31727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:54.900012 31734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:54.900002 31737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:54.900005 31735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:54.900317 31727 server_base.cc:1034] running on GCE node
I20260810 18:06:54.900839 31727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:54.901984 31727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:54.903214 31727 hybrid_clock.cc:648] HybridClock initialized: now 1786385214903191 us; error 42 us; skew 500 ppm
Aug 10 18:06:54 dist-test-slave-27sr krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385214, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:54.906437 31727 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:06:54.907560 31727 webserver.cc:460] Webserver started at http://127.21.182.66:41439/ using document root <none> and password file <none>
I20260810 18:06:54.908340 31727 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:54.908435 31727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:54.908707 31727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:54.910977 31727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "e2893b32d3904ad1a84ac18b71cb182b"
format_stamp: "Formatted at 2026-08-10 18:06:54 on dist-test-slave-27sr"
server_key: "c4dd5783d4d8030da48ba931f4ab1516"
server_key_iv: "f474d0478fdd014690b3d0df3e106ca7"
server_key_version: "encryptionkey@0"
I20260810 18:06:54.911610 31727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "e2893b32d3904ad1a84ac18b71cb182b"
format_stamp: "Formatted at 2026-08-10 18:06:54 on dist-test-slave-27sr"
server_key: "c4dd5783d4d8030da48ba931f4ab1516"
server_key_iv: "f474d0478fdd014690b3d0df3e106ca7"
server_key_version: "encryptionkey@0"
I20260810 18:06:54.916218 31727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:06:54.919188 31744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:54.920948 31727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:54.921131 31727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "e2893b32d3904ad1a84ac18b71cb182b"
format_stamp: "Formatted at 2026-08-10 18:06:54 on dist-test-slave-27sr"
server_key: "c4dd5783d4d8030da48ba931f4ab1516"
server_key_iv: "f474d0478fdd014690b3d0df3e106ca7"
server_key_version: "encryptionkey@0"
I20260810 18:06:54.921274 31727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:54.942873 31727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:54.947311 31727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:54.947570 31727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:54.948381 31727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:54.949298 31727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:54.949352 31727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:54.949397 31727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:54.949419 31727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:54.959534 31727 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:34917
I20260810 18:06:54.959556 31875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:34917 every 8 connection(s)
I20260810 18:06:54.960708 31727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:06:54.968223 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 31727
I20260810 18:06:54.968410 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:06:54.968741 22233 external_mini_cluster.cc:1428] Setting key eef77da9fef229278ea1831bde813f3c
Aug 10 18:06:54 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385214, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:54.976801 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:54.962903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:35307 (local address 127.21.182.126:35241)
0810 18:06:54.963131 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:54.963135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:54.964344 (+ 1209us) server_negotiation.cc:407] Connection header received
0810 18:06:54.965270 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:54.965273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:54.965330 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:54.965422 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:54.966735 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:54.967315 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:54.969010 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:54.969308 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:54.969404 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:54.972716 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:54.972720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:54.972723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:54.972759 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:54.974292 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:54.974901 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:54.974904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:54.974906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:54.974991 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:54.975381 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:54.975384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:54.975385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:54.975589 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:54.975701 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:54.976453 (+ 752us) server_negotiation.cc:299] Negotiation successful
0810 18:06:54.976596 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 18:06:54.977778 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:54.963341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:35307)
0810 18:06:54.964169 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:54.964229 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:54.965049 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:54.965608 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:54.965617 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:54.966016 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:54.966576 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:54.966591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:54.967501 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:54.967505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:54.968842 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:54.968855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:54.969624 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:54.969629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:54.969889 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:54.970672 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:54.970690 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:54.972519 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:54.974441 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:54.974449 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:54.974470 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:54.974768 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:54.975090 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:54.975093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:54.975095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:54.975261 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:54.975701 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:54.975711 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:54.976170 (+ 459us) client_negotiation.cc:769] Sending connection context
0810 18:06:54.976384 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 18:06:54.976770 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":124,"threads_started":1}
I20260810 18:06:54.979418 31876 heartbeater.cc:347] Connected to a master server at 127.21.182.126:35241
I20260810 18:06:54.979717 31876 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:54.980427 31876 heartbeater.cc:511] Master 127.21.182.126:35241 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
I20260810 18:06:54.981771 31499 ts_manager.cc:194] Registered new tserver with Master: e2893b32d3904ad1a84ac18b71cb182b (127.21.182.66:34917)
I20260810 18:06:54.982472 31499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:35307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:06:55.004272 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35241
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:55.128191 31881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:55.128499 31881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:55.128578 31881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:55.132781 31881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:55.132870 31881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:55.132962 31881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:55.138360 31881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:55.139515 31881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:55.140615 31881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:55.151134 31892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:55.151149 31887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:55.151156 31890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:55.151522 31881 server_base.cc:1034] running on GCE node
I20260810 18:06:55.152067 31881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:55.152799 31881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:55.153970 31881 hybrid_clock.cc:648] HybridClock initialized: now 1786385215153943 us; error 42 us; skew 500 ppm
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385215, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:55.156925 31881 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:06:55.157857 31881 webserver.cc:460] Webserver started at http://127.21.182.67:40759/ using document root <none> and password file <none>
I20260810 18:06:55.158444 31881 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:55.158505 31881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:55.158704 31881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:06:55.161091 31881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "ec15a2f7c2174215b51bd5f70ab8f7c1"
format_stamp: "Formatted at 2026-08-10 18:06:55 on dist-test-slave-27sr"
server_key: "1dce9f73ed428fc136bcea8c387b0949"
server_key_iv: "85e6d0d47bc3f208fb2575a5666e22c0"
server_key_version: "encryptionkey@0"
I20260810 18:06:55.161705 31881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "ec15a2f7c2174215b51bd5f70ab8f7c1"
format_stamp: "Formatted at 2026-08-10 18:06:55 on dist-test-slave-27sr"
server_key: "1dce9f73ed428fc136bcea8c387b0949"
server_key_iv: "85e6d0d47bc3f208fb2575a5666e22c0"
server_key_version: "encryptionkey@0"
I20260810 18:06:55.166229 31881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:06:55.169349 31899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:55.170938 31881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:55.171085 31881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "ec15a2f7c2174215b51bd5f70ab8f7c1"
format_stamp: "Formatted at 2026-08-10 18:06:55 on dist-test-slave-27sr"
server_key: "1dce9f73ed428fc136bcea8c387b0949"
server_key_iv: "85e6d0d47bc3f208fb2575a5666e22c0"
server_key_version: "encryptionkey@0"
I20260810 18:06:55.171211 31881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:55.198292 31881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:55.201944 31881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:55.202353 31881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:55.203095 31881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:55.204201 31881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:55.204259 31881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:55.204314 31881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:55.204368 31881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:55.214313 31881 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:46817
I20260810 18:06:55.214359 32023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:46817 every 8 connection(s)
I20260810 18:06:55.215335 31881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:06:55.223304 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 31881
I20260810 18:06:55.223500 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:06:55.223884 22233 external_mini_cluster.cc:1428] Setting key 37e4b559c768a5eb1c96c0a612512363
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385215, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:06:55.231033 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:55.217409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:50199 (local address 127.21.182.126:35241)
0810 18:06:55.217554 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:55.217558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:55.219026 (+ 1468us) server_negotiation.cc:407] Connection header received
0810 18:06:55.219985 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:55.219988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:55.220044 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:55.220125 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:55.221515 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.222056 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.223375 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.223690 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.223776 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:55.227172 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:55.227180 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:55.227182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:55.227211 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:55.228808 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:55.229330 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:55.229333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:55.229334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:55.229423 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:55.229817 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:55.229820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:55.229821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:55.230034 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:55.230129 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:55.230720 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 18:06:55.230901 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:06:55.231945 32027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:55.217757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:50199)
0810 18:06:55.218826 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:55.218878 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:55.219711 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:55.220311 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:55.220320 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:55.220797 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:55.221327 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:55.221342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.222178 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.222181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:55.223234 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:55.223249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.223939 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.223946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:55.224173 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:55.224887 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:55.224906 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:55.226952 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:06:55.228925 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:55.228932 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:55.228948 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:55.229232 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:55.229506 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:06:55.229508 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:06:55.229510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:06:55.229651 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:06:55.230135 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:06:55.230141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:06:55.230439 (+ 298us) client_negotiation.cc:769] Sending connection context
0810 18:06:55.230615 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 18:06:55.230915 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":568,"threads_started":1}
I20260810 18:06:55.233249 32024 heartbeater.cc:347] Connected to a master server at 127.21.182.126:35241
I20260810 18:06:55.233503 32024 heartbeater.cc:464] Registering TS with master...
I20260810 18:06:55.234052 32024 heartbeater.cc:511] Master 127.21.182.126:35241 requested a full tablet report, sending...
I20260810 18:06:55.235209 31499 ts_manager.cc:194] Registered new tserver with Master: ec15a2f7c2174215b51bd5f70ab8f7c1 (127.21.182.67:46817)
I20260810 18:06:55.235827 31499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:50199
I20260810 18:06:55.238472 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.65.keytab.
Entry for principal oryx/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.65.keytab.
WARNING: no policy specified for oryx/127.21.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.66.keytab.
Entry for principal oryx/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.66.keytab.
WARNING: no policy specified for oryx/127.21.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.67.keytab.
Entry for principal oryx/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.67.keytab.
I20260810 18:06:55.330894 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 31576
I20260810 18:06:55.341912 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 31727
I20260810 18:06:55.349992 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 31881
I20260810 18:06:55.361361 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 31466
I20260810 18:06:55.371157 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:35241
--webserver_interface=127.21.182.126
--webserver_port=34729
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:35241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:55.492825 32036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:55.493130 32036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:55.493180 32036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:55.497566 32036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:06:55.497640 32036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:55.497663 32036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:55.497681 32036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:06:55.497697 32036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:06:55.503170 32036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:35241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:35241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=34729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:55.504443 32036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:55.505549 32036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:55.514819 32044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:55.514801 32043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:55.514778 32036 server_base.cc:1034] running on GCE node
W20260810 18:06:55.514806 32046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:55.515586 32036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:55.516853 32036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:55.518046 32036 hybrid_clock.cc:648] HybridClock initialized: now 1786385215518029 us; error 28 us; skew 500 ppm
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385215, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:55.521080 32036 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:06:55.522027 32036 webserver.cc:460] Webserver started at http://127.21.182.126:34729/ using document root <none> and password file <none>
I20260810 18:06:55.522603 32036 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:55.522682 32036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:55.526707 32036 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:06:55.529263 32053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:55.530602 32036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:06:55.530759 32036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "dfcbf886c36942f59767ed1d35e88eaa"
format_stamp: "Formatted at 2026-08-10 18:06:54 on dist-test-slave-27sr"
server_key: "017876926771c8cf6ae3e3420c0097a9"
server_key_iv: "a9dd805ed3543dee847d4e64cbbaa269"
server_key_version: "encryptionkey@0"
I20260810 18:06:55.531327 32036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:55.550004 32036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:55.553402 32036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:55.553625 32036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:55.561144 32036 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:35241
I20260810 18:06:55.561174 32114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:35241 every 8 connection(s)
I20260810 18:06:55.562263 32036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:06:55.566877 32116 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:06:55.568974 32116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa: Bootstrap starting.
I20260810 18:06:55.569077 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 32036
I20260810 18:06:55.569589 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:45849
--local_ip_for_outbound_sockets=127.21.182.65
--tserver_master_addrs=127.21.182.126:35241
--webserver_port=40251
--webserver_interface=127.21.182.65
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.182.65
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:06:55.587638 32116 log.cc:824] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa: Log is configured to *not* fsync() on all Append() calls
I20260810 18:06:55.595232 32116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:06:55.595649 32116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa: Bootstrap complete.
I20260810 18:06:55.600299 32116 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbf886c36942f59767ed1d35e88eaa" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35241 } }
I20260810 18:06:55.600775 32116 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcbf886c36942f59767ed1d35e88eaa, State: Initialized, Role: FOLLOWER
I20260810 18:06:55.601353 32116 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbf886c36942f59767ed1d35e88eaa" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35241 } }
I20260810 18:06:55.601502 32116 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:06:55.601595 32116 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:06:55.601776 32116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:06:55.603855 32116 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbf886c36942f59767ed1d35e88eaa" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35241 } }
I20260810 18:06:55.604246 32116 leader_election.cc:302] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfcbf886c36942f59767ed1d35e88eaa; no voters:
I20260810 18:06:55.604609 32116 leader_election.cc:288] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:06:55.604748 32120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:06:55.604981 32120 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [term 2 LEADER]: Becoming Leader. State: Replica: dfcbf886c36942f59767ed1d35e88eaa, State: Running, Role: LEADER
I20260810 18:06:55.605365 32120 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbf886c36942f59767ed1d35e88eaa" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35241 } }
I20260810 18:06:55.605999 32116 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:06:55.606827 32121 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dfcbf886c36942f59767ed1d35e88eaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbf886c36942f59767ed1d35e88eaa" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35241 } } }
I20260810 18:06:55.606967 32121 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:55.608084 32122 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfcbf886c36942f59767ed1d35e88eaa. Latest consensus state: current_term: 2 leader_uuid: "dfcbf886c36942f59767ed1d35e88eaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcbf886c36942f59767ed1d35e88eaa" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35241 } } }
I20260810 18:06:55.608280 32122 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa [sys.catalog]: This master's current role is: LEADER
I20260810 18:06:55.608654 32127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:06:55.612310 32127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:06:55.613045 32127 catalog_manager.cc:1259] Loaded cluster ID: 3b7f3fae424d4700bc6a3d193e66811c
I20260810 18:06:55.613119 32127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:06:55.615828 32127 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:06:55.616626 32127 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dfcbf886c36942f59767ed1d35e88eaa: Loaded TSK: 0
I20260810 18:06:55.618039 32127 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:06:55.690596 32118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:55.690920 32118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:55.691012 32118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:55.695267 32118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:55.695338 32118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:55.695426 32118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:06:55.700978 32118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.65.keytab
--principal=oryx/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:45849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=40251
--tserver_master_addrs=127.21.182.126:35241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:55.702042 32118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:55.703074 32118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:55.714365 32143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:55.714377 32145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:55.714373 32142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:55.714890 32118 server_base.cc:1034] running on GCE node
I20260810 18:06:55.715330 32118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:55.716082 32118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:55.717267 32118 hybrid_clock.cc:648] HybridClock initialized: now 1786385215717227 us; error 57 us; skew 500 ppm
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385215, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:55.720317 32118 init.cc:377] Logged in from keytab as oryx/127.21.182.65@KRBTEST.COM (short username oryx)
I20260810 18:06:55.721436 32118 webserver.cc:460] Webserver started at http://127.21.182.65:40251/ using document root <none> and password file <none>
I20260810 18:06:55.722168 32118 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:55.722301 32118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:55.726920 32118 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:06:55.729396 32153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:55.730837 32118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:06:55.730981 32118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "bc763494852f4638bbf48d9dc7a19f88"
format_stamp: "Formatted at 2026-08-10 18:06:54 on dist-test-slave-27sr"
server_key: "55b1f10542ce3da70d609500652a74bf"
server_key_iv: "51f07ab6b8170c7d0cdca0204ba4d8ea"
server_key_version: "encryptionkey@0"
I20260810 18:06:55.731586 32118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:55.744663 32118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:55.748489 32118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:55.748728 32118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:55.749490 32118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:55.750480 32118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:55.750532 32118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:55.750591 32118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:55.750623 32118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:55.761719 32118 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:45849
I20260810 18:06:55.761741 32279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:45849 every 8 connection(s)
I20260810 18:06:55.762867 32118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:06:55.767815 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 32118
I20260810 18:06:55.768378 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:34917
--local_ip_for_outbound_sockets=127.21.182.66
--tserver_master_addrs=127.21.182.126:35241
--webserver_port=41439
--webserver_interface=127.21.182.66
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.182.66
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:55.777864 32283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:55.765351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:41775)
0810 18:06:55.766184 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:55.766233 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:55.767160 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:55.768553 (+ 1393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:55.768562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:55.769058 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:55.769586 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:55.769601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.771058 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.771062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:55.772105 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:55.772117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.772978 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.772983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:55.773196 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:55.773872 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:55.773893 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:55.776791 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":144,"threads_started":1}
W20260810 18:06:55.777854 32282 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:55.765221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:41775 (local address 127.21.182.126:35241)
0810 18:06:55.766011 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:55.766038 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:55.766368 (+ 330us) server_negotiation.cc:407] Connection header received
0810 18:06:55.767622 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:55.767634 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:55.768077 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:55.768462 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:55.769793 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.770894 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.772254 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.772787 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.772989 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:55.777065 (+ 4076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:41775: BlockingRecv error: recv got EOF from 127.21.182.65:41775 (error 108)
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":149,"threads_started":1}
W20260810 18:06:55.778927 32280 heartbeater.cc:644] Failed to heartbeat to 127.21.182.126:35241 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.182.126:35241: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:55.784749 32283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:55.779469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:56727)
0810 18:06:55.779609 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:55.779625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:55.779714 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:55.780029 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:55.780032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:55.780206 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:55.780310 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:55.780315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.781060 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.781063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:55.781621 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:55.781629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.782077 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.782080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:55.782171 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:55.782659 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:55.782679 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:55.784593 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 18:06:55.785099 32282 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:55.779550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:56727 (local address 127.21.182.126:35241)
0810 18:06:55.779670 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:55.779674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:55.779688 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:06:55.779840 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:55.779843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:55.779885 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:55.780013 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:55.780424 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.780940 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.781740 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.781961 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.782064 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:55.784719 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:56727: BlockingRecv error: recv got EOF from 127.21.182.65:56727 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:55.790937 32283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:55.785361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:41617)
0810 18:06:55.785499 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:55.785514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:55.785623 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:55.785872 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:55.785878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:55.786081 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:55.786179 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:55.786187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.786902 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.786905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:55.787773 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:55.787783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.788240 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.788245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:55.788340 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:55.788782 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:55.788796 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:55.790805 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 18:06:55.791075 32282 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:55.785323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:41617 (local address 127.21.182.126:35241)
0810 18:06:55.785451 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:55.785456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:55.785587 (+ 131us) server_negotiation.cc:407] Connection header received
0810 18:06:55.785711 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:55.785713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:55.785753 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:55.785842 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:55.786302 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.786785 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.787887 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.788126 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.788206 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:55.790929 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:41617: BlockingRecv error: recv got EOF from 127.21.182.65:41617 (error 108)
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":69}
W20260810 18:06:55.791388 32280 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:06:55.892239 32285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:55.892529 32285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:55.892575 32285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:55.896858 32285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:55.896924 32285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:55.896986 32285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:06:55.902541 32285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.66.keytab
--principal=oryx/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:34917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=41439
--tserver_master_addrs=127.21.182.126:35241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:55.903733 32285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:55.904842 32285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:55.915305 32294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:55.915378 32290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:55.915406 32291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:55.915673 32285 server_base.cc:1034] running on GCE node
I20260810 18:06:55.916198 32285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:55.916888 32285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:55.918073 32285 hybrid_clock.cc:648] HybridClock initialized: now 1786385215918052 us; error 39 us; skew 500 ppm
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385215, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:55.921017 32285 init.cc:377] Logged in from keytab as oryx/127.21.182.66@KRBTEST.COM (short username oryx)
I20260810 18:06:55.921979 32285 webserver.cc:460] Webserver started at http://127.21.182.66:41439/ using document root <none> and password file <none>
I20260810 18:06:55.922567 32285 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:55.922681 32285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:55.926707 32285 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 18:06:55.928987 32301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:55.930189 32285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:06:55.930323 32285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "e2893b32d3904ad1a84ac18b71cb182b"
format_stamp: "Formatted at 2026-08-10 18:06:54 on dist-test-slave-27sr"
server_key: "c4dd5783d4d8030da48ba931f4ab1516"
server_key_iv: "f474d0478fdd014690b3d0df3e106ca7"
server_key_version: "encryptionkey@0"
I20260810 18:06:55.930869 32285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:55.961413 32285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:55.964573 32285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:55.964804 32285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:55.965451 32285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:55.966349 32285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:55.966418 32285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:55.966497 32285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:55.966540 32285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:55.975647 32285 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:34917
I20260810 18:06:55.975678 32426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:34917 every 8 connection(s)
I20260810 18:06:55.976809 32285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:55.986929 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 32285
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:55.987464 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:46817
--local_ip_for_outbound_sockets=127.21.182.67
--tserver_master_addrs=127.21.182.126:35241
--webserver_port=40759
--webserver_interface=127.21.182.67
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.182.67
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:06:55.988773 32282 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:55.978868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:34047 (local address 127.21.182.126:35241)
0810 18:06:55.979053 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:55.979059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:55.980365 (+ 1306us) server_negotiation.cc:407] Connection header received
0810 18:06:55.981262 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:55.981265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:55.981321 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:55.981409 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:55.982668 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.983192 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.984464 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.984681 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.984785 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:55.988637 (+ 3852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:34047: BlockingRecv error: recv got EOF from 127.21.182.66:34047 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 18:06:55.989591 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:55.979304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:34047)
0810 18:06:55.980203 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:55.980248 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:55.981029 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:55.981617 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:55.981625 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:55.982003 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:55.982520 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:55.982534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.983336 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.983338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:55.984341 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:55.984353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.984827 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.984830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:55.984968 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:55.985545 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:55.985562 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:55.988458 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":110,"threads_started":1}
W20260810 18:06:55.990146 32427 heartbeater.cc:644] Failed to heartbeat to 127.21.182.126:35241 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.182.126:35241: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:55.996155 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:55.990665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:58683)
0810 18:06:55.990798 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:55.990813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:55.990897 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:55.991156 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:55.991159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:55.991324 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:55.991420 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:55.991424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.992214 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.992217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:55.992865 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:55.992874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.993298 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.993310 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:55.993403 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:55.993939 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:55.993950 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:55.995978 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 18:06:55.996232 32282 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:55.990740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:58683 (local address 127.21.182.126:35241)
0810 18:06:55.990899 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:55.990904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:55.990919 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:06:55.991015 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:55.991017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:55.991056 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:55.991141 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:55.991531 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.992100 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.992979 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.993188 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.993295 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:55.996097 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:58683: BlockingRecv error: recv got EOF from 127.21.182.66:58683 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:55 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:56.001595 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:55.996838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:37763)
0810 18:06:55.996948 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:55.996964 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:55.997036 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:55.997249 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:55.997251 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:55.997419 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:55.997487 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:55.997492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.998211 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.998216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:55.998726 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:55.998734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.999153 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.999157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:55.999233 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:55.999752 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:55.999763 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:56.001456 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 18:06:56.001686 32282 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:55.996938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:37763 (local address 127.21.182.126:35241)
0810 18:06:55.997053 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:55.997059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:55.997074 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:06:55.997112 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:55.997115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:55.997156 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:55.997235 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:55.997603 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.998104 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.998842 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:55.999047 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:55.999150 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:56.001567 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:37763: BlockingRecv error: recv got EOF from 127.21.182.66:37763 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 18:06:56.002036 32427 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:06:56.110597 32430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:06:56.110878 32430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:06:56.110924 32430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:06:56.115314 32430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:06:56.115394 32430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:06:56.115459 32430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:06:56.121341 32430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/krb5kdc/oryx_127.21.182.67.keytab
--principal=oryx/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:46817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=40759
--tserver_master_addrs=127.21.182.126:35241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:06:56.122550 32430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:06:56.123698 32430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:06:56.134262 32437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:56.134330 32430 server_base.cc:1034] running on GCE node
W20260810 18:06:56.134265 32436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:06:56.134267 32439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:06:56.135175 32430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:06:56.136039 32430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:06:56.137202 32430 hybrid_clock.cc:648] HybridClock initialized: now 1786385216137167 us; error 47 us; skew 500 ppm
Aug 10 18:06:56 dist-test-slave-27sr krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385216, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:06:56.140336 32430 init.cc:377] Logged in from keytab as oryx/127.21.182.67@KRBTEST.COM (short username oryx)
I20260810 18:06:56.141368 32430 webserver.cc:460] Webserver started at http://127.21.182.67:40759/ using document root <none> and password file <none>
I20260810 18:06:56.142053 32430 fs_manager.cc:359] Metadata directory not provided
I20260810 18:06:56.142141 32430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:06:56.146791 32430 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:06:56.149477 32447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:56.151324 32430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:06:56.151507 32430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "ec15a2f7c2174215b51bd5f70ab8f7c1"
format_stamp: "Formatted at 2026-08-10 18:06:55 on dist-test-slave-27sr"
server_key: "1dce9f73ed428fc136bcea8c387b0949"
server_key_iv: "85e6d0d47bc3f208fb2575a5666e22c0"
server_key_version: "encryptionkey@0"
I20260810 18:06:56.152534 32430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:06:56.168722 32430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:06:56.172616 32430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:06:56.172859 32430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:06:56.173532 32430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:06:56.174467 32430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:06:56.174541 32430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:56.174614 32430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:06:56.174656 32430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:06:56.184882 32430 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:46817
I20260810 18:06:56.184906 32575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:46817 every 8 connection(s)
I20260810 18:06:56.186010 32430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
Aug 10 18:06:56 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:56.196090 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 32430
Aug 10 18:06:56 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:06:56.198135 32282 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:56.188031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:33191 (local address 127.21.182.126:35241)
0810 18:06:56.188240 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:56.188244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:56.189451 (+ 1207us) server_negotiation.cc:407] Connection header received
0810 18:06:56.190408 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:56.190411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:56.190473 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:56.190577 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:56.191799 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.192395 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.193548 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.193785 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.193912 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:56.198026 (+ 4114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:33191: BlockingRecv error: recv got EOF from 127.21.182.67:33191 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 18:06:56.198989 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:56.188492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:33191)
0810 18:06:56.189281 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:56.189329 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:56.190174 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:56.190772 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:56.190780 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:56.191168 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:56.191652 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:56.191664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.192525 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.192529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:56.193411 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:56.193421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.193943 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.193947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:56.194133 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:56.194768 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:56.194786 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:56.197939 (+ 3153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":170,"threads_started":1}
W20260810 18:06:56.199616 32576 heartbeater.cc:644] Failed to heartbeat to 127.21.182.126:35241 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.182.126:35241: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Aug 10 18:06:56 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:56 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:56.205152 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:56.200136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:39415)
0810 18:06:56.200235 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:56.200250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:56.200351 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:56.200596 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:56.200599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:56.200769 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:56.200861 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:56.200867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.201570 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.201575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:56.202114 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:56.202122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.202575 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.202578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:56.202670 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:56.203027 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:56.203037 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:56.204993 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 18:06:56.205248 32282 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:56.200234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:39415 (local address 127.21.182.126:35241)
0810 18:06:56.200352 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:56.200357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:56.200371 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:06:56.200448 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:56.200451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:56.200491 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:56.200578 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:56.200971 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.201465 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.202235 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.202467 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.202571 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:56.205117 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:39415: BlockingRecv error: recv got EOF from 127.21.182.67:39415 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 18:06:56.207623 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:56.197267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53028 (local address 127.21.182.126:35241)
0810 18:06:56.197522 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:56.197525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:56.197537 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:06:56.197574 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:56.197576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:56.197613 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:56.197692 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:56.198612 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.199226 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.200372 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.200609 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.200726 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:56.202195 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:06:56.202202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:06:56.202226 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:06:56.202257 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:06:56.204603 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:56.205109 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:56.205116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:56.205122 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:56.205546 (+ 424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:06:56.206166 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:06:56.206168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:06:56.206169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:06:56.206668 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:06:56.206868 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 18:06:56.207162 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 18:06:56.207451 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":118,"threads_started":1}
Aug 10 18:06:56 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:56 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:56.211597 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:56.205723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:39103)
0810 18:06:56.205841 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:56.205857 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:56.205934 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:56.206121 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:56.206123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:56.206248 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:56.206412 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:56.206418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.207418 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.207423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:56.208182 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:56.208192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.208703 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.208707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:56.208793 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:56.209305 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:56.209316 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:56.211447 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 18:06:56.211676 32282 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:56.205800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:39103 (local address 127.21.182.126:35241)
0810 18:06:56.205956 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:56.205959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:56.205969 (+ 10us) server_negotiation.cc:407] Connection header received
0810 18:06:56.205999 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:56.206000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:56.206033 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:56.206094 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:56.206538 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.207265 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.208307 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.208588 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.208700 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:56.211562 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:39103: BlockingRecv error: recv got EOF from 127.21.182.67:39103 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260810 18:06:56.211951 32576 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 18:06:56 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:56 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:56.798784 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:56.792285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:50867)
0810 18:06:56.792616 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:56.792631 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:56.792797 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:56.793064 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:56.793066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:56.793261 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:56.793381 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:56.793386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.794161 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.794165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:56.794933 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:56.794943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.795584 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.795588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:56.795751 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:56.796419 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:56.796431 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:56.798630 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
W20260810 18:06:56.799011 32582 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:56.792346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:50867 (local address 127.21.182.126:35241)
0810 18:06:56.792789 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:56.792793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:56.792810 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:06:56.792890 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:56.792892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:56.792940 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:56.793031 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:56.793509 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.794053 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.795057 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:56.795438 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:56.795577 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:56.798810 (+ 3233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:50867: BlockingRecv error: recv got EOF from 127.21.182.65:50867 (error 108)
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":118,"threads_started":1}
Aug 10 18:06:57 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:57 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:57.008841 32583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:57.002763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:43873)
0810 18:06:57.003078 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:57.003098 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:57.003258 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:57.003540 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:57.003543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:57.003762 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:57.003920 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:57.003926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:57.004701 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:57.004705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:57.005297 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:57.005306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:57.005773 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:57.005777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:57.005874 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:57.006484 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:57.006497 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:57.008654 (+ 2157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":96,"threads_started":1}
W20260810 18:06:57.009006 32582 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:57.002845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:43873 (local address 127.21.182.126:35241)
0810 18:06:57.003042 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:57.003047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:57.003176 (+ 129us) server_negotiation.cc:407] Connection header received
0810 18:06:57.003369 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:57.003372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:57.003425 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:57.003513 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:57.004042 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:57.004579 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:57.005419 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:57.005647 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:57.005766 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:57.008837 (+ 3071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:43873: BlockingRecv error: recv got EOF from 127.21.182.66:43873 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 10 18:06:57 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:57 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:57.218855 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:57.212696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:44645)
0810 18:06:57.213057 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:57.213077 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:57.213190 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:57.213447 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:57.213449 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:57.213640 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:57.213738 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:57.213744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:57.214497 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:57.214499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:57.215113 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:57.215122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:57.215736 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:57.215740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:57.215833 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:57.216438 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:57.216458 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:57.218661 (+ 2203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":152,"threads_started":1}
W20260810 18:06:57.219007 32582 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:57.212839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:44645 (local address 127.21.182.126:35241)
0810 18:06:57.213006 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:57.213010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:57.213192 (+ 182us) server_negotiation.cc:407] Connection header received
0810 18:06:57.213273 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:57.213275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:57.213325 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:57.213409 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:57.213857 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:57.214396 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:57.215256 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:57.215595 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:57.215732 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:57.218883 (+ 3151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:44645: BlockingRecv error: recv got EOF from 127.21.182.67:44645 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 18:06:57 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:57 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:57.805914 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:57.799756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:46393)
0810 18:06:57.800094 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:57.800117 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:57.800242 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:57.800556 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:57.800559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:57.800751 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:57.800856 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:57.800861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:57.801815 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:57.801821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:57.802646 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:57.802659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:57.803146 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:57.803149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:57.803215 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:57.803778 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:57.803790 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:57.805782 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":80,"threads_started":1}
W20260810 18:06:57.806118 32586 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:57.799938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:46393 (local address 127.21.182.126:35241)
0810 18:06:57.800208 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:57.800211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:57.800227 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:06:57.800344 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:57.800347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:57.800395 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:57.800483 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:57.800965 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:57.801610 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:57.802784 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:57.803037 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:57.803119 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:57.805942 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:46393: BlockingRecv error: recv got EOF from 127.21.182.65:46393 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
Aug 10 18:06:58 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:58 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:58.015276 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:58.009643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:36315)
0810 18:06:58.009951 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:58.009966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:58.010078 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:58.010360 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:58.010362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:58.010547 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:58.010645 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:58.010651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:58.011432 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:58.011435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:58.012042 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:58.012052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:58.012511 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:58.012514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:58.012577 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:58.013136 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:58.013148 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:58.015145 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":89,"threads_started":1}
W20260810 18:06:58.015493 32586 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:58.009730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:36315 (local address 127.21.182.126:35241)
0810 18:06:58.009901 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:58.009905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:58.010079 (+ 174us) server_negotiation.cc:407] Connection header received
0810 18:06:58.010153 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:58.010186 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:58.010241 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:58.010327 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:58.010775 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:58.011332 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:58.012147 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:58.012384 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:58.012503 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:58.015338 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:36315: BlockingRecv error: recv got EOF from 127.21.182.66:36315 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 18:06:58 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:58 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:58.225632 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:58.219933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:56899)
0810 18:06:58.220212 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:58.220231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:58.220379 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:58.220703 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:58.220707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:58.220971 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:58.221116 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:58.221124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:58.221995 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:58.221998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:58.222555 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:58.222563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:58.223022 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:58.223024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:58.223086 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:58.223595 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:58.223607 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:58.225488 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
W20260810 18:06:58.225944 32586 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:58.220102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:56899 (local address 127.21.182.126:35241)
0810 18:06:58.220258 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:58.220262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:58.220296 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:06:58.220495 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:58.220498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:58.220558 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:58.220670 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:58.221257 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:58.221894 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:58.222691 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:58.222911 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:58.222997 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:58.225818 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:56899: BlockingRecv error: recv got EOF from 127.21.182.67:56899 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 18:06:58 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:58 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:58.813024 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:58.806887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:41467)
0810 18:06:58.807217 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:58.807233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:58.807374 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:58.807648 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:58.807651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:58.807999 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:58.808114 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:58.808121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:58.808955 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:58.808959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:58.809549 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:58.809558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:58.810084 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:58.810087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:58.810153 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:58.810760 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:58.810774 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:58.812886 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
W20260810 18:06:58.813236 32591 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:58.806995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:41467 (local address 127.21.182.126:35241)
0810 18:06:58.807259 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:58.807264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:58.807307 (+ 43us) server_negotiation.cc:407] Connection header received
0810 18:06:58.807474 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:58.807476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:58.807525 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:58.807613 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:58.808256 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:58.808825 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:58.809675 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:58.809957 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:58.810058 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:58.813056 (+ 2998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:41467: BlockingRecv error: recv got EOF from 127.21.182.65:41467 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
Aug 10 18:06:59 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:59 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:59.022910 32592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:59.016397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:57667)
0810 18:06:59.016713 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:59.016736 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:59.016849 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:59.017116 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:59.017118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:59.017336 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:59.017448 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:59.017454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:59.018250 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:59.018253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:59.018904 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:59.018913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:59.019408 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:59.019412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:59.019482 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:59.020262 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:59.020275 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:59.022763 (+ 2488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":108,"threads_started":1}
W20260810 18:06:59.023241 32591 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:59.016521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:57667 (local address 127.21.182.126:35241)
0810 18:06:59.016675 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:59.016679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:59.016814 (+ 135us) server_negotiation.cc:407] Connection header received
0810 18:06:59.016938 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:59.016940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:59.016989 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:59.017074 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:59.017572 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:59.018135 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:59.019028 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:59.019284 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:59.019373 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:59.023084 (+ 3711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:57667: BlockingRecv error: recv got EOF from 127.21.182.66:57667 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 18:06:59 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:59 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:59.233040 32593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:59.226875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:38147)
0810 18:06:59.227219 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:59.227235 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:59.227377 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:59.227709 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:59.227711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:59.227983 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:59.228137 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:59.228148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:59.228964 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:59.228969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:59.229540 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:59.229549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:59.230066 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:59.230072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:59.230158 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:59.230758 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:59.230771 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:59.232916 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":126,"threads_started":1}
W20260810 18:06:59.233220 32591 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:59.226965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:38147 (local address 127.21.182.126:35241)
0810 18:06:59.227139 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:59.227145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:59.227468 (+ 323us) server_negotiation.cc:407] Connection header received
0810 18:06:59.227534 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:59.227536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:59.227591 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:59.227672 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:59.228283 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:59.228842 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:59.229670 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:59.229917 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:59.230033 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:59.233062 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:38147: BlockingRecv error: recv got EOF from 127.21.182.67:38147 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 18:06:59 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:06:59 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:06:59.825603 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:06:59.814049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:46521)
0810 18:06:59.814496 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 18:06:59.814514 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:06:59.814646 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:06:59.814986 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:06:59.814989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:06:59.815234 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:06:59.815378 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:59.815390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:59.816349 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:59.816353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:59.817038 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:06:59.817047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:06:59.817676 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:06:59.817680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:06:59.817827 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:59.823278 (+ 5451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:06:59.823294 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:06:59.825458 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":103,"threads_started":1}
W20260810 18:06:59.825846 32595 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:06:59.814108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:46521 (local address 127.21.182.126:35241)
0810 18:06:59.814476 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0810 18:06:59.814480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:06:59.814598 (+ 118us) server_negotiation.cc:407] Connection header received
0810 18:06:59.814766 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:06:59.814770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:06:59.814842 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:06:59.814952 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:06:59.815524 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:59.816194 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:59.817254 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:06:59.817542 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:06:59.817668 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:06:59.825687 (+ 8019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:46521: BlockingRecv error: recv got EOF from 127.21.182.65:46521 (error 108)
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
Aug 10 18:07:00 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:00 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:00.029867 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:00.023784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:32797)
0810 18:07:00.024086 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:00.024100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:00.024204 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:00.024465 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:00.024467 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:00.024662 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:00.024761 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:00.024766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:00.025644 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:00.025646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:00.026229 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:00.026237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:00.026707 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:00.026710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:00.026774 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:00.027411 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:00.027423 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:00.029681 (+ 2258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":114,"threads_started":1}
W20260810 18:07:00.030071 32595 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:00.023884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:32797 (local address 127.21.182.126:35241)
0810 18:07:00.024090 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:00.024094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:00.024171 (+ 77us) server_negotiation.cc:407] Connection header received
0810 18:07:00.024292 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:00.024295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:00.024347 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:00.024427 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:00.024924 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:00.025534 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:00.026351 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:00.026595 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:00.026688 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:00.029946 (+ 3258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:32797: BlockingRecv error: recv got EOF from 127.21.182.66:32797 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 18:07:00 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:00 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:00.240350 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:00.233897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:60267)
0810 18:07:00.234339 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:00.234358 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:00.234518 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:00.234946 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:00.234950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:00.235179 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:00.235372 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:00.235382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:00.236325 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:00.236328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:00.236979 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:00.236996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:00.237473 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:00.237476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:00.237543 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:00.238128 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:00.238140 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:00.240195 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":101,"threads_started":1}
W20260810 18:07:00.240469 32595 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:00.234039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:60267 (local address 127.21.182.126:35241)
0810 18:07:00.234236 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:00.234247 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:00.234460 (+ 213us) server_negotiation.cc:407] Connection header received
0810 18:07:00.234631 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:00.234634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:00.234793 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:00.234901 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:00.235523 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:00.236215 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:00.237127 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:00.237359 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:00.237450 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:00.240338 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:60267: BlockingRecv error: recv got EOF from 127.21.182.67:60267 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 10 18:07:00 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:00 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:00.833470 32599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:00.826744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:56791)
0810 18:07:00.827046 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:00.827061 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:00.827155 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:00.827417 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:00.827419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:00.827607 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:00.827705 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:00.827710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:00.828787 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:00.828789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:00.829375 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:00.829384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:00.830054 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:00.830057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:00.830144 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:00.831013 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:00.831025 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:00.833251 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":108,"threads_started":1}
W20260810 18:07:00.833658 32600 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:00.826856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:56791 (local address 127.21.182.126:35241)
0810 18:07:00.827136 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:00.827140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:00.827157 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:07:00.827253 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:00.827256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:00.827305 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:00.827407 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:00.827969 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:00.828659 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:00.829523 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:00.829914 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:00.830046 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:00.833495 (+ 3449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:56791: BlockingRecv error: recv got EOF from 127.21.182.65:56791 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":108,"threads_started":1}
Aug 10 18:07:01 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:01 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:01.037132 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:01.030892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:42487)
0810 18:07:01.031255 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:01.031273 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:01.031390 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:01.031673 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:01.031753 (+ 80us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:01.032000 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:01.032095 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:01.032102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:01.032936 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:01.032942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:01.033654 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:01.033664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:01.034122 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:01.034126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:01.034204 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:01.034797 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:01.034817 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:01.037002 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":150,"threads_started":1}
W20260810 18:07:01.037315 32600 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:01.031084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:42487 (local address 127.21.182.126:35241)
0810 18:07:01.031256 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:01.031263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:01.031336 (+ 73us) server_negotiation.cc:407] Connection header received
0810 18:07:01.031502 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:01.031505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:01.031560 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:01.031648 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:01.032227 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:01.032830 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:01.033779 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:01.034009 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:01.034114 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:01.037149 (+ 3035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:42487: BlockingRecv error: recv got EOF from 127.21.182.66:42487 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 18:07:01 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:01 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:01.247023 32602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:01.241416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:36171)
0810 18:07:01.241693 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:01.241707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:01.241811 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:01.242090 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:01.242092 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:01.242290 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:01.242393 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:01.242398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:01.243149 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:01.243152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:01.243723 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:01.243731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:01.244256 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:01.244260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:01.244330 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:01.244913 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:01.244925 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:01.246896 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":109,"threads_started":1}
W20260810 18:07:01.247189 32600 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:01.241454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:36171 (local address 127.21.182.126:35241)
0810 18:07:01.241610 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:01.241615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:01.241783 (+ 168us) server_negotiation.cc:407] Connection header received
0810 18:07:01.241923 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:01.241926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:01.241980 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:01.242056 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:01.242493 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:01.243046 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:01.243859 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:01.244106 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:01.244204 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:01.247061 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:36171: BlockingRecv error: recv got EOF from 127.21.182.67:36171 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 18:07:01 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:01 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:01.840823 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:01.834325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:40489)
0810 18:07:01.834712 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:01.834729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:01.834882 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:01.835216 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:01.835219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:01.835546 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:01.835711 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:01.835720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:01.836879 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:01.836882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:01.837462 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:01.837471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:01.837938 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:01.837941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:01.838008 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:01.838656 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:01.838675 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:01.840693 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":100,"threads_started":1}
W20260810 18:07:01.841027 32604 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:01.834433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:40489 (local address 127.21.182.126:35241)
0810 18:07:01.834770 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:01.834774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:01.834789 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:07:01.834995 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:01.835002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:01.835060 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:01.835180 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:01.835856 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:01.836701 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:01.837597 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:01.837825 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:01.837911 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:01.840844 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:40489: BlockingRecv error: recv got EOF from 127.21.182.65:40489 (error 108)
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":94,"threads_started":1}
Aug 10 18:07:02 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:02 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:02.044875 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:02.037985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:33497)
0810 18:07:02.038277 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:02.038295 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:02.038431 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:02.038825 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:02.038828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:02.039198 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:02.039340 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:02.039351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:02.040640 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:02.040644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:02.041332 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:02.041344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:02.041853 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:02.041856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:02.041925 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:02.042470 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:02.042482 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:02.044742 (+ 2260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":110,"threads_started":1}
W20260810 18:07:02.045142 32604 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:02.038113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:33497 (local address 127.21.182.126:35241)
0810 18:07:02.038279 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:02.038283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:02.038395 (+ 112us) server_negotiation.cc:407] Connection header received
0810 18:07:02.038527 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:02.038530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:02.038583 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:02.038686 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:02.039497 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:02.040473 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:02.041458 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:02.041703 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:02.041843 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:02.045019 (+ 3176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:33497: BlockingRecv error: recv got EOF from 127.21.182.66:33497 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 18:07:02 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:02 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:02.254398 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:02.248034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:52387)
0810 18:07:02.248304 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:02.248322 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:02.248462 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:02.248783 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:02.248786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:02.249085 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:02.249216 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:02.249227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:02.250161 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:02.250164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:02.250730 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:02.250739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:02.251224 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:02.251227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:02.251291 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:02.251877 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:02.251929 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:02.254168 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":91,"threads_started":1}
W20260810 18:07:02.254446 32604 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:02.248086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:52387 (local address 127.21.182.126:35241)
0810 18:07:02.248291 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:02.248297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:02.248439 (+ 142us) server_negotiation.cc:407] Connection header received
0810 18:07:02.248561 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:02.248564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:02.248633 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:02.248748 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:02.249367 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:02.250041 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:02.250875 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:02.251114 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:02.251204 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:02.254324 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:52387: BlockingRecv error: recv got EOF from 127.21.182.67:52387 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 10 18:07:02 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:02 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:02.847661 32607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:02.841653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:44963)
0810 18:07:02.842025 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:02.842039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:02.842138 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:02.842413 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:02.842415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:02.842613 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:02.842719 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:02.842724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:02.843519 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:02.843522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:02.844146 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:02.844158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:02.844753 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:02.844757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:02.844827 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:02.845396 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:02.845409 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:02.847534 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":137,"threads_started":1}
W20260810 18:07:02.847852 32608 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:02.841796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:44963 (local address 127.21.182.126:35241)
0810 18:07:02.842079 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:02.842083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:02.842099 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:07:02.842235 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:02.842237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:02.842287 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:02.842374 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:02.842843 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:02.843382 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:02.844301 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:02.844541 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:02.844634 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:02.847697 (+ 3063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:44963: BlockingRecv error: recv got EOF from 127.21.182.65:44963 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":104,"threads_started":1}
Aug 10 18:07:03 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:03 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:03.052559 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:03.045863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:50515)
0810 18:07:03.046233 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:03.046248 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:03.046400 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:03.046665 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:03.046667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:03.046878 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:03.046977 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:03.046983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:03.048067 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:03.048072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:03.048824 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:03.048837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:03.049333 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:03.049336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:03.049407 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:03.050020 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:03.050032 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:03.052419 (+ 2387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
W20260810 18:07:03.052927 32608 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:03.046038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:50515 (local address 127.21.182.126:35241)
0810 18:07:03.046208 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:03.046212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:03.046340 (+ 128us) server_negotiation.cc:407] Connection header received
0810 18:07:03.046502 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:03.046505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:03.046552 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:03.046620 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:03.047130 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:03.047931 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:03.048952 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:03.049180 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:03.049278 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:03.052618 (+ 3340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:50515: BlockingRecv error: recv got EOF from 127.21.182.66:50515 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 18:07:03 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:03 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:03.262573 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:03.255574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:33433)
0810 18:07:03.256004 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:03.256027 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:03.256204 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:03.256579 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:03.256585 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:03.256837 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:03.256943 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:03.256949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:03.258066 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:03.258069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:03.258757 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:03.258767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:03.259300 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:03.259304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:03.259393 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:03.260075 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:03.260089 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:03.262427 (+ 2338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":111,"threads_started":1}
W20260810 18:07:03.262734 32608 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:03.255751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:33433 (local address 127.21.182.126:35241)
0810 18:07:03.255968 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:03.255973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:03.256175 (+ 202us) server_negotiation.cc:407] Connection header received
0810 18:07:03.256354 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:03.256357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:03.256425 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:03.256513 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:03.257090 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:03.257916 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:03.258896 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:03.259123 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:03.259214 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:03.262629 (+ 3415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:33433: BlockingRecv error: recv got EOF from 127.21.182.67:33433 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 10 18:07:03 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:03 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:03.856386 32611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:03.849027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:54165)
0810 18:07:03.849359 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:03.849378 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:03.849546 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:03.849996 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:03.849999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:03.850280 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:03.850473 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:03.850482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:03.851612 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:03.851616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:03.852446 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:03.852459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:03.853048 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:03.853052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:03.853131 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:03.853796 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:03.853811 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:03.856111 (+ 2300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
W20260810 18:07:03.856478 32612 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:03.849215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:54165 (local address 127.21.182.126:35241)
0810 18:07:03.849492 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:03.849506 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:03.849524 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:07:03.849766 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:03.849768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:03.849820 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:03.849908 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:03.850832 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:03.851466 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:03.852603 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:03.852888 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:03.852983 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:03.856331 (+ 3348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:54165: BlockingRecv error: recv got EOF from 127.21.182.65:54165 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
Aug 10 18:07:04 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:04 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:04.060698 32613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:04.053784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:59865)
0810 18:07:04.054109 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:04.054133 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:04.054255 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:04.054567 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:04.054569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:04.054790 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:04.054903 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:04.054909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:04.055964 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:04.055969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:04.056623 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:04.056648 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:04.057184 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:04.057188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:04.057278 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:04.057880 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:04.057893 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:04.060568 (+ 2675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":118,"threads_started":1}
W20260810 18:07:04.060973 32612 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:04.053871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:59865 (local address 127.21.182.126:35241)
0810 18:07:04.054136 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:04.054139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:04.054215 (+ 76us) server_negotiation.cc:407] Connection header received
0810 18:07:04.054380 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:04.054382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:04.054432 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:04.054512 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:04.055039 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:04.055759 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:04.056774 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:04.057021 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:04.057165 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:04.060714 (+ 3549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:59865: BlockingRecv error: recv got EOF from 127.21.182.66:59865 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Aug 10 18:07:04 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:04 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:04.270797 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:04.263993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:50753)
0810 18:07:04.264307 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:04.264323 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:04.264462 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:04.264783 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:04.264786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:04.265040 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:04.265146 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:04.265152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:04.266106 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:04.266110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:04.266826 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:04.266838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:04.267409 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:04.267412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:04.267486 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:04.268144 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:04.268158 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:04.270478 (+ 2320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":83,"threads_started":1}
W20260810 18:07:04.270807 32612 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:04.264091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:50753 (local address 127.21.182.126:35241)
0810 18:07:04.264331 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:04.264335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:04.264493 (+ 158us) server_negotiation.cc:407] Connection header received
0810 18:07:04.264557 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:04.264559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:04.264617 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:04.264702 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:04.265312 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:04.265951 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:04.266994 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:04.267260 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:04.267378 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:04.270680 (+ 3302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:50753: BlockingRecv error: recv got EOF from 127.21.182.67:50753 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Aug 10 18:07:04 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:04 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:04.871003 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:04.857411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:36013)
0810 18:07:04.857895 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:04.857913 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:04.858028 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:04.858375 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:04.858378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:04.858576 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:04.858696 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:04.858702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:04.859529 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:04.859532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:04.860250 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:04.860265 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:04.860825 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:04.860830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:04.860922 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:04.868082 (+ 7160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:04.868103 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:04.870875 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":123,"threads_started":1}
W20260810 18:07:04.871279 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:04.857567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:36013 (local address 127.21.182.126:35241)
0810 18:07:04.857958 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:04.857963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:04.857977 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:07:04.858164 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:04.858166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:04.858227 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:04.858313 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:04.858838 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:04.859401 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:04.860420 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:04.860680 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:04.860799 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:04.871053 (+ 10254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:36013: BlockingRecv error: recv got EOF from 127.21.182.65:36013 (error 108)
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":107,"threads_started":1}
Aug 10 18:07:05 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:05 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:05.069017 32618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:05.061837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:43875)
0810 18:07:05.062220 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:05.062234 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:05.062392 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:05.062830 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:05.062834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:05.063086 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:05.063206 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:05.063213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:05.064168 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:05.064173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:05.064832 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:05.064841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:05.065397 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:05.065401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:05.065504 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:05.066157 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:05.066191 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:05.068827 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":135,"threads_started":1}
W20260810 18:07:05.069111 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:05.061892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:43875 (local address 127.21.182.126:35241)
0810 18:07:05.062109 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:05.062114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:05.062327 (+ 213us) server_negotiation.cc:407] Connection header received
0810 18:07:05.062558 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:05.062564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:05.062622 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:05.062795 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:05.063350 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:05.064032 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:05.064972 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:05.065252 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:05.065384 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:05.068984 (+ 3600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:43875: BlockingRecv error: recv got EOF from 127.21.182.66:43875 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 18:07:05 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:05 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:05.278934 32619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:05.271973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:37369)
0810 18:07:05.272345 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:05.272361 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:05.272524 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:05.272856 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:05.272860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:05.273120 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:05.273223 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:05.273230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:05.274144 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:05.274151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:05.274785 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:05.274794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:05.275311 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:05.275315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:05.275423 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:05.276192 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:05.276205 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:05.278717 (+ 2512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":129,"threads_started":1}
W20260810 18:07:05.279157 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:05.272054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:37369 (local address 127.21.182.126:35241)
0810 18:07:05.272253 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:05.272259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:05.272454 (+ 195us) server_negotiation.cc:407] Connection header received
0810 18:07:05.272648 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:05.272651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:05.272716 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:05.272870 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:05.273369 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:05.274011 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:05.274925 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:05.275177 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:05.275304 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:05.279019 (+ 3715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:37369: BlockingRecv error: recv got EOF from 127.21.182.67:37369 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 18:07:05 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:05 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:05.878356 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:05.872167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:55987)
0810 18:07:05.872524 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:05.872540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:05.872655 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:05.873045 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:05.873047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:05.873244 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:05.873348 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:05.873353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:05.874148 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:05.874151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:05.874865 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:05.874878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:05.875473 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:05.875477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:05.875557 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:05.876247 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:05.876260 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:05.878233 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":128,"threads_started":1}
W20260810 18:07:05.878566 32621 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:05.872382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:55987 (local address 127.21.182.126:35241)
0810 18:07:05.872622 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:05.872626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:05.872642 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:07:05.872866 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:05.872869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:05.872917 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:05.873004 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:05.873456 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:05.874029 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:05.874993 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:05.875338 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:05.875425 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:05.878382 (+ 2957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:55987: BlockingRecv error: recv got EOF from 127.21.182.65:55987 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
Aug 10 18:07:06 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:06 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:06.076798 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:06.070230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:47767)
0810 18:07:06.070613 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:06.070634 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:06.070893 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:06.071215 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:06.071219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:06.071461 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:06.071581 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:06.071587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:06.072513 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:06.072519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:06.073172 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:06.073181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:06.073711 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:06.073716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:06.073838 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:06.074381 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:06.074395 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:06.076646 (+ 2251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":108,"threads_started":1}
W20260810 18:07:06.077090 32621 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:06.070393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:47767 (local address 127.21.182.126:35241)
0810 18:07:06.070548 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:06.070552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:06.070814 (+ 262us) server_negotiation.cc:407] Connection header received
0810 18:07:06.071016 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:06.071021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:06.071082 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:06.071196 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:06.071722 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:06.072388 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:06.073308 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:06.073573 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:06.073700 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:06.076831 (+ 3131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:47767: BlockingRecv error: recv got EOF from 127.21.182.66:47767 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 18:07:06 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:06 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:06.286317 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:06.280055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:34203)
0810 18:07:06.280539 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:06.280563 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:06.280710 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:06.281009 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:06.281012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:06.281237 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:06.281339 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:06.281345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:06.282127 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:06.282132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:06.282739 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:06.282748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:06.283239 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:06.283243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:06.283331 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:06.283994 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:06.284006 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:06.286183 (+ 2177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":115,"threads_started":1}
W20260810 18:07:06.286520 32621 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:06.280247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:34203 (local address 127.21.182.126:35241)
0810 18:07:06.280501 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:06.280507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:06.280652 (+ 145us) server_negotiation.cc:407] Connection header received
0810 18:07:06.280819 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:06.280822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:06.280883 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:06.280976 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:06.281473 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:06.282011 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:06.282879 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:06.283114 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:06.283229 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:06.286354 (+ 3125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:34203: BlockingRecv error: recv got EOF from 127.21.182.67:34203 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
Aug 10 18:07:06 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:06 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:06.885622 32625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:06.879429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:40193)
0810 18:07:06.879841 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:06.879860 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:06.880005 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:06.880341 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:06.880344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:06.880569 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:06.880671 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:06.880677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:06.881508 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:06.881514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:06.882096 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:06.882105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:06.882610 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:06.882613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:06.882704 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:06.883359 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:06.883372 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:06.885417 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":124,"threads_started":1}
W20260810 18:07:06.885728 32624 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:06.879487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:40193 (local address 127.21.182.126:35241)
0810 18:07:06.880089 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:06.880096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:06.880111 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:07:06.880159 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:06.880161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:06.880202 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:06.880303 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:06.880839 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:06.881387 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:06.882231 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:06.882481 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:06.882600 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:06.885577 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:40193: BlockingRecv error: recv got EOF from 127.21.182.65:40193 (error 108)
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":104,"threads_started":1}
Aug 10 18:07:07 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:07 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:07.084008 32626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:07.077699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:52783)
0810 18:07:07.078002 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:07.078017 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:07.078127 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:07.078423 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:07.078425 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:07.078631 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:07.078815 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:07.078824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:07.079682 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:07.079685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:07.080426 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:07.080437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:07.080943 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:07.080946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:07.081047 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:07.081629 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:07.081642 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:07.083730 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
W20260810 18:07:07.084035 32624 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:07.077770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:52783 (local address 127.21.182.126:35241)
0810 18:07:07.077925 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:07.077929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:07.078092 (+ 163us) server_negotiation.cc:407] Connection header received
0810 18:07:07.078219 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:07.078222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:07.078269 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:07.078377 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:07.078970 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:07.079534 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:07.080570 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:07.080816 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:07.080913 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:07.083930 (+ 3017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:52783: BlockingRecv error: recv got EOF from 127.21.182.66:52783 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 18:07:07 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:07 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:07.293272 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:07.287231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:54353)
0810 18:07:07.287551 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:07.287565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:07.287671 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:07.287964 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:07.287967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:07.288244 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:07.288355 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:07.288362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:07.289174 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:07.289177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:07.289816 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:07.289825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:07.290274 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:07.290277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:07.290349 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:07.290874 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:07.290886 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:07.293134 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":129,"threads_started":1}
W20260810 18:07:07.293502 32624 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:07.287281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:54353 (local address 127.21.182.126:35241)
0810 18:07:07.287442 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:07.287447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:07.287642 (+ 195us) server_negotiation.cc:407] Connection header received
0810 18:07:07.287747 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:07.287749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:07.287798 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:07.287883 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:07.288486 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:07.289033 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:07.289927 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:07.290142 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:07.290221 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:07.293341 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:54353: BlockingRecv error: recv got EOF from 127.21.182.67:54353 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 18:07:07 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:07 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:07.893471 32628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:07.886550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:38269)
0810 18:07:07.886881 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:07.886897 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:07.887003 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:07.887458 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:07.887461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:07.887687 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:07.887807 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:07.887814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:07.888822 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:07.888825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:07.889469 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:07.889479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:07.890084 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:07.890087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:07.890184 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:07.890859 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:07.890873 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:07.893254 (+ 2381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":95,"threads_started":1}
W20260810 18:07:07.893538 32629 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:07.886755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:38269 (local address 127.21.182.126:35241)
0810 18:07:07.886985 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:07.886989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:07.887002 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:07:07.887107 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:07.887109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:07.887153 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:07.887441 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:07.887997 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:07.888620 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:07.889606 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:07.889950 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:07.890047 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:07.893396 (+ 3349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:38269: BlockingRecv error: recv got EOF from 127.21.182.65:38269 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
Aug 10 18:07:08 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:08 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:08.091873 32630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:08.085189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:38273)
0810 18:07:08.085602 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:08.085616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:08.085843 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:08.086133 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:08.086135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:08.086362 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:08.086458 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:08.086464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:08.087259 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:08.087265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:08.087838 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:08.087847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:08.088611 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:08.088615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:08.088715 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:08.089300 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:08.089312 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:08.091693 (+ 2381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":145,"threads_started":1}
W20260810 18:07:08.091988 32629 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:08.085318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:38273 (local address 127.21.182.126:35241)
0810 18:07:08.085544 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:08.085548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:08.085775 (+ 227us) server_negotiation.cc:407] Connection header received
0810 18:07:08.085967 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:08.085970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:08.086019 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:08.086108 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:08.086583 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:08.087122 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:08.088016 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:08.088427 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:08.088596 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:08.091835 (+ 3239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:38273: BlockingRecv error: recv got EOF from 127.21.182.66:38273 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 10 18:07:08 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:08 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:08.301002 32632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:08.294328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:34959)
0810 18:07:08.294690 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:08.294782 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:08.294914 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:08.295195 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:08.295197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:08.295407 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:08.295513 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:08.295518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:08.296343 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:08.296346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:08.297019 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:08.297031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:08.297711 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:08.297717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:08.297822 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:08.298498 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:08.298515 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:08.300788 (+ 2273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":137,"threads_started":1}
W20260810 18:07:08.301031 32629 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:08.294445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:34959 (local address 127.21.182.126:35241)
0810 18:07:08.294622 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:08.294630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:08.294870 (+ 240us) server_negotiation.cc:407] Connection header received
0810 18:07:08.295006 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:08.295009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:08.295060 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:08.295143 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:08.295624 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:08.296201 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:08.297155 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:08.297530 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:08.297659 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:08.300926 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:34959: BlockingRecv error: recv got EOF from 127.21.182.67:34959 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 18:07:08 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:08 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:08.901062 32634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:08.894633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:51205)
0810 18:07:08.894982 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:08.894998 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:08.895120 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:08.895425 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:08.895428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:08.895636 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:08.895751 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:08.895756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:08.896919 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:08.896922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:08.897779 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:08.897790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:08.898254 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:08.898256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:08.898317 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:08.898864 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:08.898877 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:08.900943 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":102,"threads_started":1}
W20260810 18:07:08.901288 32635 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:08.894799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:51205 (local address 127.21.182.126:35241)
0810 18:07:08.895051 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:08.895055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:08.895071 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:07:08.895239 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:08.895242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:08.895291 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:08.895378 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:08.895876 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:08.896759 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:08.897892 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:08.898120 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:08.898212 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:08.901099 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:51205: BlockingRecv error: recv got EOF from 127.21.182.65:51205 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":65,"threads_started":1}
Aug 10 18:07:09 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:09 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:09.099022 32637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:09.093071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:51141)
0810 18:07:09.093399 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:09.093413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:09.093525 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:09.093785 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:09.093787 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:09.093985 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:09.094091 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:09.094096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:09.094906 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:09.094910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:09.095527 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:09.095536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:09.096056 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:09.096059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:09.096132 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:09.096684 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:09.096700 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:09.098876 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":114,"threads_started":1}
W20260810 18:07:09.099246 32635 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:09.093222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:51141 (local address 127.21.182.126:35241)
0810 18:07:09.093396 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:09.093400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:09.093495 (+ 95us) server_negotiation.cc:407] Connection header received
0810 18:07:09.093597 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:09.093600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:09.093646 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:09.093722 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:09.094222 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:09.094767 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:09.095645 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:09.095861 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:09.096017 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:09.099056 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:51141: BlockingRecv error: recv got EOF from 127.21.182.66:51141 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 18:07:09 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:09 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:09.308043 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:09.301880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:58011)
0810 18:07:09.302296 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:09.302349 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:09.302504 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:09.302806 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:09.302809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:09.303077 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:09.303173 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:09.303180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:09.303997 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:09.304003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:09.304574 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:09.304582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:09.305072 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:09.305075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:09.305161 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:09.305747 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:09.305759 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:09.307841 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":91,"threads_started":1}
W20260810 18:07:09.308185 32635 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:09.301957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:58011 (local address 127.21.182.126:35241)
0810 18:07:09.302241 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:09.302246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:09.302433 (+ 187us) server_negotiation.cc:407] Connection header received
0810 18:07:09.302610 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:09.302615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:09.302673 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:09.302774 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:09.303304 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:09.303845 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:09.304719 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:09.304947 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:09.305055 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:09.308065 (+ 3010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:58011: BlockingRecv error: recv got EOF from 127.21.182.67:58011 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
Aug 10 18:07:09 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:09 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:09.914628 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:09.901846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:50213)
0810 18:07:09.902241 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:09.902262 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:09.902387 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:09.902678 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:09.902683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:09.902961 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:09.903114 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:09.903123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:09.904108 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:09.904114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:09.905148 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:09.905163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:09.905720 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:09.905726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:09.905838 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:09.911882 (+ 6044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:09.911928 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:09.914444 (+ 2516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":133,"threads_started":1}
W20260810 18:07:09.914902 32640 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:09.901977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:50213 (local address 127.21.182.126:35241)
0810 18:07:09.902304 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:09.902307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:09.902322 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:07:09.902496 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:09.902498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:09.902545 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:09.902624 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:09.903325 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:09.903972 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:09.905294 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:09.905592 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:09.905677 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:09.914593 (+ 8916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:50213: BlockingRecv error: recv got EOF from 127.21.182.65:50213 (error 108)
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":72,"threads_started":1}
Aug 10 18:07:10 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:10 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:10.106123 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:10.100087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:44557)
0810 18:07:10.100428 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:10.100443 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:10.100559 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:10.100874 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:10.100877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:10.101098 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:10.101208 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:10.101214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:10.102042 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:10.102045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:10.102743 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:10.102754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:10.103232 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:10.103239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:10.103306 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:10.103985 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:10.103998 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:10.105993 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":93,"threads_started":1}
W20260810 18:07:10.106331 32640 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:10.100183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:44557 (local address 127.21.182.126:35241)
0810 18:07:10.100423 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:10.100427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:10.100506 (+ 79us) server_negotiation.cc:407] Connection header received
0810 18:07:10.100665 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:10.100667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:10.100717 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:10.100795 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:10.101361 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:10.101911 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:10.102876 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:10.103117 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:10.103205 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:10.106181 (+ 2976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:44557: BlockingRecv error: recv got EOF from 127.21.182.66:44557 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Aug 10 18:07:10 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:10 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.67@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:10.314603 32644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:10.309033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.67:34585)
0810 18:07:10.309324 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:10.309338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:10.309450 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:10.309706 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:10.309708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:10.309906 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:10.310002 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:10.310008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:10.310761 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:10.310763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:10.311374 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:10.311384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:10.311823 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:10.311826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:10.311886 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:10.312512 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:10.312524 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:10.314464 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
W20260810 18:07:10.314742 32640 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:10.309128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:34585 (local address 127.21.182.126:35241)
0810 18:07:10.309334 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:10.309338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:10.309417 (+ 79us) server_negotiation.cc:407] Connection header received
0810 18:07:10.309537 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:10.309539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:10.309589 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:10.309664 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:10.310112 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:10.310658 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:10.311487 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:10.311716 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:10.311811 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:10.314631 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.67:34585: BlockingRecv error: recv got EOF from 127.21.182.67:34585 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 10 18:07:10 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:10 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.65@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:10.921482 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:10.915617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.65:44663)
0810 18:07:10.915981 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:10.916007 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:10.916111 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:10.916375 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:10.916377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:10.916584 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:10.916687 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:10.916692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:10.917473 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:10.917476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:10.918130 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:10.918140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:10.918625 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:10.918627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:10.918711 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:10.919252 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:10.919265 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:10.921339 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":84,"threads_started":1}
W20260810 18:07:10.921643 32647 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:10.915695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:44663 (local address 127.21.182.126:35241)
0810 18:07:10.916066 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:10.916069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:10.916084 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:07:10.916199 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:10.916202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:10.916244 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:10.916356 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:10.916797 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:10.917360 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:10.918267 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:10.918518 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:10.918600 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:10.921501 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.65:44663: BlockingRecv error: recv got EOF from 127.21.182.65:44663 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":115,"threads_started":1}
Aug 10 18:07:11 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:07:11 dist-test-slave-27sr krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.182.66@KRBTEST.COM for oryx/127.21.182.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:07:11.113708 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:11.107343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35241 (local address 127.21.182.66:55925)
0810 18:07:11.107800 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:11.107819 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:11.107974 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:11.108389 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:11.108393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:11.108630 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:11.108796 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:11.108808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:11.109641 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:11.109643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:11.110220 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:11.110229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:11.110709 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:11.110712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:11.110774 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:11.111357 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:11.111370 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:11.113578 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.182.126:35241: Server oryx/127.21.182.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":107,"threads_started":1}
W20260810 18:07:11.114006 32647 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:11.107411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:55925 (local address 127.21.182.126:35241)
0810 18:07:11.107607 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:11.107614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:11.107950 (+ 336us) server_negotiation.cc:407] Connection header received
0810 18:07:11.108165 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:11.108169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:11.108244 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:11.108357 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:11.108940 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:11.109525 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:11.110352 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:11.110596 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:11.110688 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:11.113874 (+ 3186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.182.66:55925: BlockingRecv error: recv got EOF from 127.21.182.66:55925 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 18:07:11.204417 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 32118
I20260810 18:07:11.215996 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 32285
I20260810 18:07:11.224330 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 32430
I20260810 18:07:11.238437 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 32036
2026-08-10T18:07:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18329 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T18:07:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:07:11Z Disabled control of system clock
I20260810 18:07:11.275185 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:39459
--webserver_interface=127.21.182.126
--webserver_port=0
--builtin_ntp_servers=127.21.182.84:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:39459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 18:07:11.393487 32652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:07:11.393766 32652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:07:11.393802 32652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:07:11.397722 32652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:07:11.397782 32652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:07:11.397797 32652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:07:11.397815 32652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:07:11.397836 32652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:07:11.402654 32652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:39459
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:39459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:07:11.403808 32652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:07:11.405342 32652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:07:11.414480 32657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:11.414464 32660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:11.414464 32658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:11.414526 32652 server_base.cc:1034] running on GCE node
I20260810 18:07:11.415297 32652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:07:11.416478 32652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:07:11.417678 32652 hybrid_clock.cc:648] HybridClock initialized: now 1786385231417666 us; error 27 us; skew 500 ppm
I20260810 18:07:11.419669 32652 webserver.cc:460] Webserver started at http://127.21.182.126:35903/ using document root <none> and password file <none>
I20260810 18:07:11.420444 32652 fs_manager.cc:359] Metadata directory not provided
I20260810 18:07:11.420542 32652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:07:11.420790 32652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:07:11.422912 32652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "889f80fb82a445f59750da1070c8e544"
format_stamp: "Formatted at 2026-08-10 18:07:11 on dist-test-slave-27sr"
server_key: "47e48f1e5362c9f50ae625f9cfb3685e"
server_key_iv: "0d352750e2dfe5f2bab7a1f8513ffc78"
server_key_version: "encryptionkey@0"
I20260810 18:07:11.423542 32652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "889f80fb82a445f59750da1070c8e544"
format_stamp: "Formatted at 2026-08-10 18:07:11 on dist-test-slave-27sr"
server_key: "47e48f1e5362c9f50ae625f9cfb3685e"
server_key_iv: "0d352750e2dfe5f2bab7a1f8513ffc78"
server_key_version: "encryptionkey@0"
I20260810 18:07:11.427532 32652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 18:07:11.430213 32666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:11.431559 32652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:07:11.431712 32652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "889f80fb82a445f59750da1070c8e544"
format_stamp: "Formatted at 2026-08-10 18:07:11 on dist-test-slave-27sr"
server_key: "47e48f1e5362c9f50ae625f9cfb3685e"
server_key_iv: "0d352750e2dfe5f2bab7a1f8513ffc78"
server_key_version: "encryptionkey@0"
I20260810 18:07:11.431847 32652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:07:11.442116 32652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:07:11.442912 32652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:07:11.443137 32652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:07:11.450984 32652 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:39459
I20260810 18:07:11.451009 32725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:39459 every 8 connection(s)
I20260810 18:07:11.452090 32652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:07:11.452309 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 32652
I20260810 18:07:11.452435 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:07:11.452917 22233 external_mini_cluster.cc:1428] Setting key 6dcea5347948e3df20cc0fd3e5994274
I20260810 18:07:11.455714 32726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:07:11.462548 32726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544: Bootstrap starting.
I20260810 18:07:11.462962 32729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:11.454384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.21.182.126:39459)
0810 18:07:11.455046 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:11.455063 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:11.455100 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:07:11.455775 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:11.455791 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:11.456299 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:11.456596 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:11.456973 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:11.457959 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:11.459236 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:11.459724 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:11.459878 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:11.460855 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:07:11.460860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:07:11.460874 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:07:11.461285 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:07:11.461342 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:07:11.461362 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:07:11.461458 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:11.461670 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 18:07:11.462042 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":140,"threads_started":1}
I20260810 18:07:11.465204 32726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544: Neither blocks nor log segments found. Creating new log.
I20260810 18:07:11.466075 32726 log.cc:824] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544: Log is configured to *not* fsync() on all Append() calls
I20260810 18:07:11.468428 32726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544: No bootstrap required, opened a new log
I20260810 18:07:11.472664 32726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889f80fb82a445f59750da1070c8e544" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39459 } }
I20260810 18:07:11.472949 32726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:07:11.472986 32726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889f80fb82a445f59750da1070c8e544, State: Initialized, Role: FOLLOWER
I20260810 18:07:11.473389 32726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889f80fb82a445f59750da1070c8e544" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39459 } }
I20260810 18:07:11.473510 32726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:07:11.473558 32726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:07:11.473631 32726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:07:11.474632 32726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889f80fb82a445f59750da1070c8e544" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39459 } }
I20260810 18:07:11.474980 32726 leader_election.cc:302] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 889f80fb82a445f59750da1070c8e544; no voters:
I20260810 18:07:11.475281 32726 leader_election.cc:288] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:07:11.475525 32732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:07:11.475930 32732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [term 1 LEADER]: Becoming Leader. State: Replica: 889f80fb82a445f59750da1070c8e544, State: Running, Role: LEADER
I20260810 18:07:11.476352 32732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889f80fb82a445f59750da1070c8e544" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39459 } }
I20260810 18:07:11.476819 32726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:07:11.478562 32734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "889f80fb82a445f59750da1070c8e544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889f80fb82a445f59750da1070c8e544" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39459 } } }
I20260810 18:07:11.478562 32735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 889f80fb82a445f59750da1070c8e544. Latest consensus state: current_term: 1 leader_uuid: "889f80fb82a445f59750da1070c8e544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889f80fb82a445f59750da1070c8e544" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39459 } } }
I20260810 18:07:11.478796 32735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [sys.catalog]: This master's current role is: LEADER
I20260810 18:07:11.478796 32734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544 [sys.catalog]: This master's current role is: LEADER
I20260810 18:07:11.479377 32749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:07:11.482883 32749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:07:11.489256 32749 catalog_manager.cc:1347] Generated new cluster ID: 28b4ffaa6fc7408984cde5bb128e1a24
I20260810 18:07:11.489358 32749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:07:11.496788 32749 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:07:11.497727 32749 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:07:11.502903 32749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 889f80fb82a445f59750da1070c8e544: Generated new TSK 0
I20260810 18:07:11.503786 32749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:07:11.513413 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39459
--builtin_ntp_servers=127.21.182.84:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 18:07:11.633520 32756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:07:11.633993 32756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:07:11.634052 32756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:07:11.639283 32756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:07:11.639353 32756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:07:11.639415 32756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:07:11.644721 32756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:07:11.646067 32756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:07:11.647615 32756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:07:11.658823 32762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:11.658829 32761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:11.658823 32764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:11.659181 32756 server_base.cc:1034] running on GCE node
I20260810 18:07:11.659647 32756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:07:11.660424 32756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:07:11.661623 32756 hybrid_clock.cc:648] HybridClock initialized: now 1786385231661598 us; error 47 us; skew 500 ppm
I20260810 18:07:11.663655 32756 webserver.cc:460] Webserver started at http://127.21.182.65:35149/ using document root <none> and password file <none>
I20260810 18:07:11.664393 32756 fs_manager.cc:359] Metadata directory not provided
I20260810 18:07:11.664495 32756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:07:11.664758 32756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:07:11.667054 32756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "4d307efd3c4d417d8e3d16d0833ab187"
format_stamp: "Formatted at 2026-08-10 18:07:11 on dist-test-slave-27sr"
server_key: "f154def7e500a4dd9fd3e7c7227a1105"
server_key_iv: "ccb0c9c5e28c0b176bdfb6e3acfa1d1c"
server_key_version: "encryptionkey@0"
I20260810 18:07:11.667766 32756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "4d307efd3c4d417d8e3d16d0833ab187"
format_stamp: "Formatted at 2026-08-10 18:07:11 on dist-test-slave-27sr"
server_key: "f154def7e500a4dd9fd3e7c7227a1105"
server_key_iv: "ccb0c9c5e28c0b176bdfb6e3acfa1d1c"
server_key_version: "encryptionkey@0"
I20260810 18:07:11.672405 32756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:07:11.675534 302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:11.677627 32756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:07:11.677812 32756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "4d307efd3c4d417d8e3d16d0833ab187"
format_stamp: "Formatted at 2026-08-10 18:07:11 on dist-test-slave-27sr"
server_key: "f154def7e500a4dd9fd3e7c7227a1105"
server_key_iv: "ccb0c9c5e28c0b176bdfb6e3acfa1d1c"
server_key_version: "encryptionkey@0"
I20260810 18:07:11.677958 32756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:07:11.702028 32756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:07:11.702826 32756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:07:11.703037 32756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:07:11.703706 32756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:07:11.704780 32756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:07:11.704855 32756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:11.704923 32756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:07:11.704972 32756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:11.714414 32756 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:45851
I20260810 18:07:11.714434 433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:45851 every 8 connection(s)
I20260810 18:07:11.715382 32756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:07:11.722512 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 32756
I20260810 18:07:11.722657 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:07:11.723062 22233 external_mini_cluster.cc:1428] Setting key db7ef4ddcf2a8ef7b5f9cded08503b2f
I20260810 18:07:11.725456 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39459
--builtin_ntp_servers=127.21.182.84:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 18:07:11.726267 32729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:11.717433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:60477 (local address 127.21.182.126:39459)
0810 18:07:11.717582 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:11.717586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:11.718752 (+ 1166us) server_negotiation.cc:407] Connection header received
0810 18:07:11.720212 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:11.720216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:11.720278 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:11.720360 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:11.721295 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:11.721832 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:11.723742 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:11.724074 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:11.724199 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:11.725608 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:07:11.725613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:07:11.725618 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:07:11.725637 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:07:11.725743 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:07:11.725757 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:07:11.725830 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:11.726042 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 18:07:11.726114 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:07:11.727440 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:11.717819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:39459 (local address 127.21.182.65:60477)
0810 18:07:11.718500 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:11.718542 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:11.719990 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:11.720541 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:11.720547 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:11.720629 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:11.721137 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:11.721150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:11.721952 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:11.721955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:11.723528 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:11.723546 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:11.724253 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:11.724256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:11.724420 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:11.725163 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:07:11.725165 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:11.725301 (+ 136us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:07:11.725396 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:07:11.725859 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:07:11.725873 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:07:11.725882 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:07:11.726024 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 18:07:11.726355 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260810 18:07:11.728789 434 heartbeater.cc:347] Connected to a master server at 127.21.182.126:39459
I20260810 18:07:11.729071 434 heartbeater.cc:464] Registering TS with master...
I20260810 18:07:11.729707 434 heartbeater.cc:511] Master 127.21.182.126:39459 requested a full tablet report, sending...
I20260810 18:07:11.731644 32686 ts_manager.cc:194] Registered new tserver with Master: 4d307efd3c4d417d8e3d16d0833ab187 (127.21.182.65:45851)
I20260810 18:07:11.733496 32686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.182.65:60477
W20260810 18:07:11.846412 437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:07:11.846701 437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:07:11.846740 437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:07:11.850719 437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:07:11.850783 437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:07:11.850836 437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:07:11.855829 437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:07:11.856973 437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:07:11.858337 437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:07:11.868310 443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:11.868322 442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:11.868358 446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:11.868525 437 server_base.cc:1034] running on GCE node
I20260810 18:07:11.869010 437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:07:11.869652 437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:07:11.870894 437 hybrid_clock.cc:648] HybridClock initialized: now 1786385231870857 us; error 50 us; skew 500 ppm
I20260810 18:07:11.873041 437 webserver.cc:460] Webserver started at http://127.21.182.66:43963/ using document root <none> and password file <none>
I20260810 18:07:11.873708 437 fs_manager.cc:359] Metadata directory not provided
I20260810 18:07:11.873811 437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:07:11.874061 437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:07:11.876271 437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "ef92a5e6317142c0b718a2e5af684ec0"
format_stamp: "Formatted at 2026-08-10 18:07:11 on dist-test-slave-27sr"
server_key: "1c58cd725b37509b3a7b66978bf71ef2"
server_key_iv: "a3bf6f0afc68fa51d51c2b395c7b1ba3"
server_key_version: "encryptionkey@0"
I20260810 18:07:11.876940 437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "ef92a5e6317142c0b718a2e5af684ec0"
format_stamp: "Formatted at 2026-08-10 18:07:11 on dist-test-slave-27sr"
server_key: "1c58cd725b37509b3a7b66978bf71ef2"
server_key_iv: "a3bf6f0afc68fa51d51c2b395c7b1ba3"
server_key_version: "encryptionkey@0"
I20260810 18:07:11.881165 437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 18:07:11.884040 454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:11.885372 437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:07:11.885509 437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "ef92a5e6317142c0b718a2e5af684ec0"
format_stamp: "Formatted at 2026-08-10 18:07:11 on dist-test-slave-27sr"
server_key: "1c58cd725b37509b3a7b66978bf71ef2"
server_key_iv: "a3bf6f0afc68fa51d51c2b395c7b1ba3"
server_key_version: "encryptionkey@0"
I20260810 18:07:11.885627 437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:07:11.904060 437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:07:11.904853 437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:07:11.905117 437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:07:11.905753 437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:07:11.906765 437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:07:11.906838 437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:11.906913 437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:07:11.906963 437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:11.916460 437 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:42213
I20260810 18:07:11.916505 585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:42213 every 8 connection(s)
I20260810 18:07:11.917474 437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:07:11.924513 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 437
I20260810 18:07:11.924659 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:07:11.924966 22233 external_mini_cluster.cc:1428] Setting key 3672e758711d7ab110514cbda1dd34d8
I20260810 18:07:11.927198 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39459
--builtin_ntp_servers=127.21.182.84:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 18:07:11.927644 32729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:11.919610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:40649 (local address 127.21.182.126:39459)
0810 18:07:11.919827 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:11.919831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:11.920930 (+ 1099us) server_negotiation.cc:407] Connection header received
0810 18:07:11.922378 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:11.922381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:11.922431 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:11.922509 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:11.923418 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:11.923964 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:11.925267 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:11.925579 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:11.925684 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:11.927019 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:07:11.927023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:07:11.927025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:07:11.927037 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:07:11.927056 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:07:11.927064 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:07:11.927145 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:11.927361 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 18:07:11.927435 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260810 18:07:11.928628 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:11.920052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:39459 (local address 127.21.182.66:40649)
0810 18:07:11.920779 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:11.920821 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:11.922151 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:11.922682 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:11.922689 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:11.922772 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:11.923288 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:11.923302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:11.924074 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:11.924077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:11.925144 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:11.925154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:11.925742 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:11.925747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:11.925963 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:11.926726 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:07:11.926728 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:11.926783 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:07:11.926851 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:07:11.927149 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:07:11.927160 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:07:11.927169 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:07:11.927329 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 18:07:11.927624 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":121,"threads_started":1}
I20260810 18:07:11.930029 586 heartbeater.cc:347] Connected to a master server at 127.21.182.126:39459
I20260810 18:07:11.930305 586 heartbeater.cc:464] Registering TS with master...
I20260810 18:07:11.930881 586 heartbeater.cc:511] Master 127.21.182.126:39459 requested a full tablet report, sending...
I20260810 18:07:11.932330 32686 ts_manager.cc:194] Registered new tserver with Master: ef92a5e6317142c0b718a2e5af684ec0 (127.21.182.66:42213)
I20260810 18:07:11.933238 32686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.182.66:40649
W20260810 18:07:12.049059 589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:07:12.049364 589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:07:12.049402 589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:07:12.053642 589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:07:12.053726 589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:07:12.053785 589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:07:12.058867 589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:07:12.060196 589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:07:12.061671 589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:07:12.072880 595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:12.072952 589 server_base.cc:1034] running on GCE node
W20260810 18:07:12.072880 598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:12.072880 594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:12.073743 589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:07:12.074633 589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:07:12.075883 589 hybrid_clock.cc:648] HybridClock initialized: now 1786385232075841 us; error 60 us; skew 500 ppm
I20260810 18:07:12.078521 589 webserver.cc:460] Webserver started at http://127.21.182.67:33871/ using document root <none> and password file <none>
I20260810 18:07:12.079336 589 fs_manager.cc:359] Metadata directory not provided
I20260810 18:07:12.079460 589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:07:12.079710 589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:07:12.082227 589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "9bb8d78fd2384a1db91ff0d202921145"
format_stamp: "Formatted at 2026-08-10 18:07:12 on dist-test-slave-27sr"
server_key: "a4c1f8c592b00f00cbb7f6012341afc5"
server_key_iv: "88630a6885543dbe85a34a1f33d4a239"
server_key_version: "encryptionkey@0"
I20260810 18:07:12.082891 589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "9bb8d78fd2384a1db91ff0d202921145"
format_stamp: "Formatted at 2026-08-10 18:07:12 on dist-test-slave-27sr"
server_key: "a4c1f8c592b00f00cbb7f6012341afc5"
server_key_iv: "88630a6885543dbe85a34a1f33d4a239"
server_key_version: "encryptionkey@0"
I20260810 18:07:12.087183 589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 18:07:12.090353 606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:12.092213 589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 18:07:12.092384 589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "9bb8d78fd2384a1db91ff0d202921145"
format_stamp: "Formatted at 2026-08-10 18:07:12 on dist-test-slave-27sr"
server_key: "a4c1f8c592b00f00cbb7f6012341afc5"
server_key_iv: "88630a6885543dbe85a34a1f33d4a239"
server_key_version: "encryptionkey@0"
I20260810 18:07:12.092545 589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:07:12.117067 589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:07:12.117920 589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:07:12.118152 589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:07:12.118862 589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:07:12.119982 589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:07:12.120059 589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:12.120136 589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:07:12.120179 589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:12.130149 589 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:35277
I20260810 18:07:12.130172 732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:35277 every 8 connection(s)
I20260810 18:07:12.131224 589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:07:12.135736 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 589
I20260810 18:07:12.135857 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:07:12.136242 22233 external_mini_cluster.cc:1428] Setting key 8eebd2efb89a252ae19ddc2b096b85ef
I20260810 18:07:12.142661 32729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:12.133493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:49045 (local address 127.21.182.126:39459)
0810 18:07:12.133658 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:12.133662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:12.134922 (+ 1260us) server_negotiation.cc:407] Connection header received
0810 18:07:12.136726 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:12.136734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:12.136811 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:12.136899 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:12.138297 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:12.139238 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:12.140494 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:12.140741 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:12.140838 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:12.142090 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:07:12.142093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:07:12.142095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:07:12.142110 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:07:12.142133 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:07:12.142142 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:07:12.142206 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:12.142443 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 18:07:12.142522 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 18:07:12.143741 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:12.133971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:39459 (local address 127.21.182.67:49045)
0810 18:07:12.134731 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:12.134792 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:12.136486 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:12.137164 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:12.137176 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:12.137362 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:12.138137 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:12.138162 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:12.139379 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:12.139382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:12.140336 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:12.140346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:12.140901 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:12.140904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:12.141066 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:12.141778 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:07:12.141780 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:12.141837 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:07:12.141916 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:07:12.142223 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:07:12.142266 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:07:12.142271 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 18:07:12.142390 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 18:07:12.142669 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":143,"threads_started":1}
I20260810 18:07:12.145238 733 heartbeater.cc:347] Connected to a master server at 127.21.182.126:39459
I20260810 18:07:12.145576 733 heartbeater.cc:464] Registering TS with master...
I20260810 18:07:12.146236 733 heartbeater.cc:511] Master 127.21.182.126:39459 requested a full tablet report, sending...
I20260810 18:07:12.147568 32686 ts_manager.cc:194] Registered new tserver with Master: 9bb8d78fd2384a1db91ff0d202921145 (127.21.182.67:35277)
I20260810 18:07:12.148312 32686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.182.67:49045
I20260810 18:07:12.151597 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 18:07:12.154989 32729 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:12.153895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.21.182.126:39459)
0810 18:07:12.154043 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:12.154047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:12.154262 (+ 215us) server_negotiation.cc:407] Connection header received
0810 18:07:12.154367 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:12.154370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:12.154415 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:12.154493 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:12.154759 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52776: BlockingRecv error: recv got EOF from 127.0.0.1:52776 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:07:12.155261 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 32756
I20260810 18:07:12.172901 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 437
I20260810 18:07:12.187381 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 589
I20260810 18:07:12.195052 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 32652
2026-08-10T18:07:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (955 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T18:07:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:07:12Z Disabled control of system clock
I20260810 18:07:12.231058 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:37055
--webserver_interface=127.21.182.126
--webserver_port=0
--builtin_ntp_servers=127.21.182.84:39327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:37055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 18:07:12.352699 747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:07:12.353016 747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:07:12.353056 747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:07:12.357251 747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:07:12.357381 747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:07:12.357446 747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:07:12.357489 747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:07:12.357525 747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:07:12.362720 747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:39327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:37055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:37055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:07:12.363981 747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:07:12.365511 747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:07:12.375020 755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:12.374994 753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:12.374993 757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:12.375525 747 server_base.cc:1034] running on GCE node
I20260810 18:07:12.376415 747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:07:12.377779 747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:07:12.379012 747 hybrid_clock.cc:648] HybridClock initialized: now 1786385232378977 us; error 55 us; skew 500 ppm
I20260810 18:07:12.381476 747 webserver.cc:460] Webserver started at http://127.21.182.126:44339/ using document root <none> and password file <none>
I20260810 18:07:12.382155 747 fs_manager.cc:359] Metadata directory not provided
I20260810 18:07:12.382259 747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:07:12.382511 747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:07:12.384847 747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "70d2782027d344b3a46c43e14583dce8"
format_stamp: "Formatted at 2026-08-10 18:07:12 on dist-test-slave-27sr"
server_key: "52014336290bdde734599db759f5d0dd"
server_key_iv: "743b25c601383a634885169ec49b2a81"
server_key_version: "encryptionkey@0"
I20260810 18:07:12.385453 747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "70d2782027d344b3a46c43e14583dce8"
format_stamp: "Formatted at 2026-08-10 18:07:12 on dist-test-slave-27sr"
server_key: "52014336290bdde734599db759f5d0dd"
server_key_iv: "743b25c601383a634885169ec49b2a81"
server_key_version: "encryptionkey@0"
I20260810 18:07:12.389545 747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:07:12.392534 763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:12.393985 747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:07:12.394155 747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "70d2782027d344b3a46c43e14583dce8"
format_stamp: "Formatted at 2026-08-10 18:07:12 on dist-test-slave-27sr"
server_key: "52014336290bdde734599db759f5d0dd"
server_key_iv: "743b25c601383a634885169ec49b2a81"
server_key_version: "encryptionkey@0"
I20260810 18:07:12.394290 747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:07:12.399499 747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:07:12.400307 747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:07:12.400534 747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:07:12.408090 747 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:37055
I20260810 18:07:12.408099 821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:37055 every 8 connection(s)
I20260810 18:07:12.409191 747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:07:12.412825 822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:07:12.418736 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 747
I20260810 18:07:12.418902 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:07:12.419024 822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8: Bootstrap starting.
I20260810 18:07:12.419241 22233 external_mini_cluster.cc:1428] Setting key 782b691c0321f7cd1e73b79d73dffaf7
I20260810 18:07:12.422082 822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8: Neither blocks nor log segments found. Creating new log.
I20260810 18:07:12.423079 822 log.cc:824] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8: Log is configured to *not* fsync() on all Append() calls
I20260810 18:07:12.425668 822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8: No bootstrap required, opened a new log
I20260810 18:07:12.426016 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:12.420881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59216 (local address 127.21.182.126:37055)
0810 18:07:12.421667 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:12.421689 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:12.421734 (+ 45us) server_negotiation.cc:407] Connection header received
0810 18:07:12.422695 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:12.422723 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:12.423232 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:12.423634 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:12.424552 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:07:12.424557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:07:12.424570 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:07:12.424580 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:07:12.424638 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:07:12.424655 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:07:12.424750 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:12.424903 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 18:07:12.425196 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":126,"threads_started":1}
I20260810 18:07:12.430051 822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d2782027d344b3a46c43e14583dce8" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37055 } }
I20260810 18:07:12.430339 822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:07:12.430434 822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d2782027d344b3a46c43e14583dce8, State: Initialized, Role: FOLLOWER
I20260810 18:07:12.430897 822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d2782027d344b3a46c43e14583dce8" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37055 } }
I20260810 18:07:12.431053 822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:07:12.431145 822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:07:12.431277 822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:07:12.432399 822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d2782027d344b3a46c43e14583dce8" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37055 } }
I20260810 18:07:12.432765 822 leader_election.cc:302] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70d2782027d344b3a46c43e14583dce8; no voters:
I20260810 18:07:12.433172 822 leader_election.cc:288] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:07:12.433290 830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:07:12.433516 830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [term 1 LEADER]: Becoming Leader. State: Replica: 70d2782027d344b3a46c43e14583dce8, State: Running, Role: LEADER
I20260810 18:07:12.433849 830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d2782027d344b3a46c43e14583dce8" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37055 } }
I20260810 18:07:12.434581 822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:07:12.435734 831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70d2782027d344b3a46c43e14583dce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d2782027d344b3a46c43e14583dce8" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37055 } } }
I20260810 18:07:12.435873 831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [sys.catalog]: This master's current role is: LEADER
I20260810 18:07:12.436201 832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d2782027d344b3a46c43e14583dce8. Latest consensus state: current_term: 1 leader_uuid: "70d2782027d344b3a46c43e14583dce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d2782027d344b3a46c43e14583dce8" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 37055 } } }
I20260810 18:07:12.436288 832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8 [sys.catalog]: This master's current role is: LEADER
I20260810 18:07:12.436337 840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:07:12.439863 840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:07:12.446194 840 catalog_manager.cc:1347] Generated new cluster ID: 4f0d50340f574265ae55a40d397fcb90
I20260810 18:07:12.446285 840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:07:12.453953 840 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:07:12.454316 840 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:07:12.466672 840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70d2782027d344b3a46c43e14583dce8: Generated new TSK 0
I20260810 18:07:12.467679 840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:07:12.471457 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37055
--builtin_ntp_servers=127.21.182.84:39327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 18:07:12.610844 852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:07:12.611188 852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:07:12.611269 852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:07:12.616161 852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:07:12.616250 852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:07:12.616351 852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:07:12.621923 852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:39327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:07:12.623200 852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:07:12.624826 852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:07:12.636044 859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:12.636164 852 server_base.cc:1034] running on GCE node
W20260810 18:07:12.636039 861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:12.636421 858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:12.636958 852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:07:12.637634 852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:07:12.638800 852 hybrid_clock.cc:648] HybridClock initialized: now 1786385232638783 us; error 35 us; skew 500 ppm
I20260810 18:07:12.641099 852 webserver.cc:460] Webserver started at http://127.21.182.65:36721/ using document root <none> and password file <none>
I20260810 18:07:12.641783 852 fs_manager.cc:359] Metadata directory not provided
I20260810 18:07:12.641852 852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:07:12.642143 852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:07:12.644505 852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "91aa7a9bc6a546fcb93b5bd137065bbe"
format_stamp: "Formatted at 2026-08-10 18:07:12 on dist-test-slave-27sr"
server_key: "f3a040d8dfdab9ba55e3efc0071a2213"
server_key_iv: "6d69c26ec6199b1deba645d11de1ef5d"
server_key_version: "encryptionkey@0"
I20260810 18:07:12.645198 852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "91aa7a9bc6a546fcb93b5bd137065bbe"
format_stamp: "Formatted at 2026-08-10 18:07:12 on dist-test-slave-27sr"
server_key: "f3a040d8dfdab9ba55e3efc0071a2213"
server_key_iv: "6d69c26ec6199b1deba645d11de1ef5d"
server_key_version: "encryptionkey@0"
I20260810 18:07:12.649379 852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:07:12.652289 867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:12.653702 852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:07:12.653812 852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "91aa7a9bc6a546fcb93b5bd137065bbe"
format_stamp: "Formatted at 2026-08-10 18:07:12 on dist-test-slave-27sr"
server_key: "f3a040d8dfdab9ba55e3efc0071a2213"
server_key_iv: "6d69c26ec6199b1deba645d11de1ef5d"
server_key_version: "encryptionkey@0"
I20260810 18:07:12.653913 852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:07:12.658814 852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:07:12.659440 852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:07:12.659590 852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:07:12.660247 852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:07:12.661255 852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:07:12.661319 852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:12.661361 852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:07:12.661377 852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:12.670778 852 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:37013
I20260810 18:07:12.670815 1000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:37013 every 8 connection(s)
I20260810 18:07:12.671851 852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:07:12.679054 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:12.674055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:35565 (local address 127.21.182.126:37055)
0810 18:07:12.674246 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:12.674250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:12.675646 (+ 1396us) server_negotiation.cc:407] Connection header received
0810 18:07:12.677160 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:12.677163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:12.677213 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:12.677298 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:12.678506 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:07:12.678509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:07:12.678512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:07:12.678534 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:07:12.678555 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:07:12.678564 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:07:12.678688 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:12.678850 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 18:07:12.678902 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 18:07:12.680069 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:12.674520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37055 (local address 127.21.182.65:35565)
0810 18:07:12.675447 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:12.675530 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:12.676911 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:12.677493 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:12.677501 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:12.677603 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:12.678195 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:07:12.678197 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:12.678240 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:07:12.678336 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:07:12.678664 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:07:12.678674 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:07:12.678681 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:07:12.678882 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 18:07:12.679150 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":149,"threads_started":1}
I20260810 18:07:12.681499 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 852
I20260810 18:07:12.681540 1001 heartbeater.cc:347] Connected to a master server at 127.21.182.126:37055
I20260810 18:07:12.681672 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:07:12.681872 1001 heartbeater.cc:464] Registering TS with master...
I20260810 18:07:12.682138 22233 external_mini_cluster.cc:1428] Setting key d98a6af2f5f093907fc9c5ea2d300839
I20260810 18:07:12.682384 1001 heartbeater.cc:511] Master 127.21.182.126:37055 requested a full tablet report, sending...
I20260810 18:07:12.684216 782 ts_manager.cc:194] Registered new tserver with Master: 91aa7a9bc6a546fcb93b5bd137065bbe (127.21.182.65:37013)
I20260810 18:07:12.684480 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37055
--builtin_ntp_servers=127.21.182.84:39327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 18:07:12.808461 1005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:07:12.808820 1005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:07:12.808895 1005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:07:12.813654 1005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:07:12.813760 1005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:07:12.813870 1005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:07:12.819162 1005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:39327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:07:12.820573 1005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:07:12.822202 1005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:07:12.833423 1010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:12.833403 1005 server_base.cc:1034] running on GCE node
W20260810 18:07:12.833391 1013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:12.833376 1011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:12.834414 1005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:07:12.835214 1005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:07:12.836426 1005 hybrid_clock.cc:648] HybridClock initialized: now 1786385232836406 us; error 50 us; skew 500 ppm
I20260810 18:07:12.838991 1005 webserver.cc:460] Webserver started at http://127.21.182.66:45135/ using document root <none> and password file <none>
I20260810 18:07:12.839720 1005 fs_manager.cc:359] Metadata directory not provided
I20260810 18:07:12.839802 1005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:07:12.840111 1005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:07:12.842753 1005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "9f59b8f9ff61402697650a8da56df1b2"
format_stamp: "Formatted at 2026-08-10 18:07:12 on dist-test-slave-27sr"
server_key: "ca55db36dd8f604a2de4525d8aec66a4"
server_key_iv: "d9daa17c2496cea8531dc40628aa00e8"
server_key_version: "encryptionkey@0"
I20260810 18:07:12.843461 1005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "9f59b8f9ff61402697650a8da56df1b2"
format_stamp: "Formatted at 2026-08-10 18:07:12 on dist-test-slave-27sr"
server_key: "ca55db36dd8f604a2de4525d8aec66a4"
server_key_iv: "d9daa17c2496cea8531dc40628aa00e8"
server_key_version: "encryptionkey@0"
I20260810 18:07:12.848153 1005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 18:07:12.851239 1019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:12.852906 1005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:07:12.853077 1005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "9f59b8f9ff61402697650a8da56df1b2"
format_stamp: "Formatted at 2026-08-10 18:07:12 on dist-test-slave-27sr"
server_key: "ca55db36dd8f604a2de4525d8aec66a4"
server_key_iv: "d9daa17c2496cea8531dc40628aa00e8"
server_key_version: "encryptionkey@0"
I20260810 18:07:12.853215 1005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:07:12.858572 1005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:07:12.859320 1005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:07:12.859532 1005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:07:12.860211 1005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:07:12.861263 1005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:07:12.861337 1005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:12.861410 1005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:07:12.861462 1005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:12.871564 1005 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:39999
I20260810 18:07:12.871635 1146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:39999 every 8 connection(s)
I20260810 18:07:12.872840 1005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:07:12.880882 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:12.875422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:43813 (local address 127.21.182.126:37055)
0810 18:07:12.875598 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:12.875602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:12.877048 (+ 1446us) server_negotiation.cc:407] Connection header received
0810 18:07:12.878690 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:12.878694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:12.878752 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:12.878868 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:12.880245 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:07:12.880249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:07:12.880252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:07:12.880277 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:07:12.880304 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:07:12.880323 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:07:12.880468 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:12.880646 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 18:07:12.880704 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 18:07:12.881992 1149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:12.875980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37055 (local address 127.21.182.66:43813)
0810 18:07:12.876842 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:12.876892 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:12.878392 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:12.879085 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:12.879093 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:12.879207 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:12.879881 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:07:12.879884 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:12.879948 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:07:12.880038 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:07:12.880445 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:07:12.880457 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:07:12.880464 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:07:12.880687 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 18:07:12.880980 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":148,"threads_started":1}
I20260810 18:07:12.882922 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 1005
I20260810 18:07:12.883081 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:07:12.883490 22233 external_mini_cluster.cc:1428] Setting key e07ff11cf7a54a6007ce7877a0c64c8e
I20260810 18:07:12.883610 1147 heartbeater.cc:347] Connected to a master server at 127.21.182.126:37055
I20260810 18:07:12.883913 1147 heartbeater.cc:464] Registering TS with master...
I20260810 18:07:12.884500 1147 heartbeater.cc:511] Master 127.21.182.126:37055 requested a full tablet report, sending...
I20260810 18:07:12.885792 782 ts_manager.cc:194] Registered new tserver with Master: 9f59b8f9ff61402697650a8da56df1b2 (127.21.182.66:39999)
I20260810 18:07:12.886132 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37055
--builtin_ntp_servers=127.21.182.84:39327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 18:07:13.017115 1150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:07:13.017462 1150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:07:13.017534 1150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:07:13.022070 1150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:07:13.022187 1150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:07:13.022286 1150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:07:13.027616 1150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:39327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:37055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:07:13.029309 1150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:07:13.030951 1150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:07:13.042536 1155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:13.042521 1156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:13.042521 1158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:13.042997 1150 server_base.cc:1034] running on GCE node
I20260810 18:07:13.043475 1150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:07:13.044292 1150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:07:13.045461 1150 hybrid_clock.cc:648] HybridClock initialized: now 1786385233045424 us; error 49 us; skew 500 ppm
I20260810 18:07:13.047564 1150 webserver.cc:460] Webserver started at http://127.21.182.67:41323/ using document root <none> and password file <none>
I20260810 18:07:13.048300 1150 fs_manager.cc:359] Metadata directory not provided
I20260810 18:07:13.048372 1150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:07:13.048647 1150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:07:13.050956 1150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "87b1dc57dc1e43ed9f016e27972083c9"
format_stamp: "Formatted at 2026-08-10 18:07:13 on dist-test-slave-27sr"
server_key: "7e22870d429688f9ff5cb033b64091a6"
server_key_iv: "cbbc28feba3981777eb8ae406c8bce0d"
server_key_version: "encryptionkey@0"
I20260810 18:07:13.051589 1150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "87b1dc57dc1e43ed9f016e27972083c9"
format_stamp: "Formatted at 2026-08-10 18:07:13 on dist-test-slave-27sr"
server_key: "7e22870d429688f9ff5cb033b64091a6"
server_key_iv: "cbbc28feba3981777eb8ae406c8bce0d"
server_key_version: "encryptionkey@0"
I20260810 18:07:13.055771 1150 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:07:13.059193 1164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:13.060837 1150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:07:13.061005 1150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "87b1dc57dc1e43ed9f016e27972083c9"
format_stamp: "Formatted at 2026-08-10 18:07:13 on dist-test-slave-27sr"
server_key: "7e22870d429688f9ff5cb033b64091a6"
server_key_iv: "cbbc28feba3981777eb8ae406c8bce0d"
server_key_version: "encryptionkey@0"
I20260810 18:07:13.061167 1150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:07:13.066362 1150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:07:13.067085 1150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:07:13.067298 1150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:07:13.067983 1150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:07:13.069021 1150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:07:13.069070 1150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:13.069139 1150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:07:13.069172 1150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:13.079304 1150 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:46499
I20260810 18:07:13.079335 1288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:46499 every 8 connection(s)
I20260810 18:07:13.080487 1150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:07:13.084625 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 1150
I20260810 18:07:13.084743 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:07:13.085090 22233 external_mini_cluster.cc:1428] Setting key 5408ad2768bca2d3d5769a199c6abb8c
I20260810 18:07:13.087926 827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:13.082677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:60861 (local address 127.21.182.126:37055)
0810 18:07:13.082824 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:13.082829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:13.084096 (+ 1267us) server_negotiation.cc:407] Connection header received
0810 18:07:13.085711 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:13.085715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:13.085770 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:13.085884 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:13.087368 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:07:13.087374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:07:13.087379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:07:13.087393 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:07:13.087422 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:07:13.087430 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:07:13.087525 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:13.087707 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 18:07:13.087759 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:07:13.088894 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:13.083087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:37055 (local address 127.21.182.67:60861)
0810 18:07:13.083872 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:13.083975 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:13.085419 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:13.086125 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:13.086133 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:13.086278 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:13.087026 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:07:13.087027 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:13.087072 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:07:13.087172 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:07:13.087527 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:07:13.087538 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:07:13.087547 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:07:13.087754 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 18:07:13.088073 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":146,"threads_started":1}
I20260810 18:07:13.090031 1289 heartbeater.cc:347] Connected to a master server at 127.21.182.126:37055
I20260810 18:07:13.090314 1289 heartbeater.cc:464] Registering TS with master...
I20260810 18:07:13.090771 1289 heartbeater.cc:511] Master 127.21.182.126:37055 requested a full tablet report, sending...
I20260810 18:07:13.092016 782 ts_manager.cc:194] Registered new tserver with Master: 87b1dc57dc1e43ed9f016e27972083c9 (127.21.182.67:46499)
I20260810 18:07:13.100122 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 18:07:13.103730 827 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:07:13.102586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59226 (local address 127.21.182.126:37055)
0810 18:07:13.102749 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:13.102755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:13.102998 (+ 243us) server_negotiation.cc:407] Connection header received
0810 18:07:13.103159 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:13.103162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:13.103211 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:13.103300 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:13.103583 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59226: BlockingRecv error: recv got EOF from 127.0.0.1:59226 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:07:13.103916 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 852
I20260810 18:07:13.116752 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 1005
I20260810 18:07:13.128360 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 1150
I20260810 18:07:13.137669 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 747
2026-08-10T18:07:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (943 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:07:13 dist-test-slave-27sr krb5kdc[1299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:07:13 dist-test-slave-27sr krb5kdc[1299](info): set up 2 sockets
Aug 10 18:07:13 dist-test-slave-27sr krb5kdc[1299](info): commencing operation
krb5kdc: starting...
W20260810 18:07:14.322587 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:07:14 dist-test-slave-27sr krb5kdc[1299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:07:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:07:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:07:14.469197 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:39553
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:39553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:07:14.589782 1314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:07:14.590071 1314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:07:14.590114 1314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:07:14.594112 1314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:07:14.594175 1314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:07:14.594195 1314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:07:14.594213 1314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:07:14.594228 1314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:07:14.599584 1314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:39553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:39553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:07:14.600716 1314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:07:14.601715 1314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:07:14.610924 1320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:14.610924 1319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:14.611020 1322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:14.611112 1314 server_base.cc:1034] running on GCE node
I20260810 18:07:14.611963 1314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:07:14.613193 1314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:07:14.614369 1314 hybrid_clock.cc:648] HybridClock initialized: now 1786385234614335 us; error 50 us; skew 500 ppm
Aug 10 18:07:14 dist-test-slave-27sr krb5kdc[1299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385234, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:07:14.617394 1314 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:07:14.618398 1314 webserver.cc:460] Webserver started at http://127.21.182.126:44125/ using document root <none> and password file <none>
I20260810 18:07:14.619009 1314 fs_manager.cc:359] Metadata directory not provided
I20260810 18:07:14.619102 1314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:07:14.619361 1314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:07:14.621625 1314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "251019eade1748b7a0e68e697a2a494e"
format_stamp: "Formatted at 2026-08-10 18:07:14 on dist-test-slave-27sr"
server_key: "2914aab8d2e404eadd3a5c15e4cc8427"
server_key_iv: "660a227fd66309c302cd120dcd1edd72"
server_key_version: "encryptionkey@0"
I20260810 18:07:14.622223 1314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "251019eade1748b7a0e68e697a2a494e"
format_stamp: "Formatted at 2026-08-10 18:07:14 on dist-test-slave-27sr"
server_key: "2914aab8d2e404eadd3a5c15e4cc8427"
server_key_iv: "660a227fd66309c302cd120dcd1edd72"
server_key_version: "encryptionkey@0"
I20260810 18:07:14.626227 1314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:07:14.629092 1332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:14.630505 1314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:07:14.630667 1314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "251019eade1748b7a0e68e697a2a494e"
format_stamp: "Formatted at 2026-08-10 18:07:14 on dist-test-slave-27sr"
server_key: "2914aab8d2e404eadd3a5c15e4cc8427"
server_key_iv: "660a227fd66309c302cd120dcd1edd72"
server_key_version: "encryptionkey@0"
I20260810 18:07:14.630798 1314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:07:14.651823 1314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:07:14.655061 1314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:07:14.655295 1314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:07:14.663461 1314 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:39553
I20260810 18:07:14.663494 1393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:39553 every 8 connection(s)
I20260810 18:07:14.664705 1314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:07:14.667557 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 1314
I20260810 18:07:14.667709 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:07:14.667870 1394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:07:14.668097 22233 external_mini_cluster.cc:1428] Setting key 033e8092f8ce2ec0f710763fcee6ae0d
I20260810 18:07:14.675064 1394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e: Bootstrap starting.
Aug 10 18:07:14 dist-test-slave-27sr krb5kdc[1299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:07:14.678457 1394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e: Neither blocks nor log segments found. Creating new log.
I20260810 18:07:14.679581 1394 log.cc:824] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:07:14.682371 1394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e: No bootstrap required, opened a new log
I20260810 18:07:14.683961 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:14.669597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.21.182.126:39553)
0810 18:07:14.670098 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:14.670114 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:14.670154 (+ 40us) server_negotiation.cc:407] Connection header received
0810 18:07:14.670948 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:14.670970 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:14.671327 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:14.671811 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:14.672735 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:14.673775 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:14.674687 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:14.675198 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:14.675394 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:14.678451 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:07:14.678458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:07:14.678480 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:07:14.678523 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:07:14.680539 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:07:14.681128 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:07:14.681135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:07:14.681141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:07:14.681285 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:07:14.681582 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:07:14.681585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:07:14.681587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:07:14.682148 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:07:14.682359 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:14.682740 (+ 381us) server_negotiation.cc:299] Negotiation successful
0810 18:07:14.683088 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":162,"threads_started":1}
I20260810 18:07:14.686947 1394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251019eade1748b7a0e68e697a2a494e" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39553 } }
I20260810 18:07:14.687179 1394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:07:14.687258 1394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 251019eade1748b7a0e68e697a2a494e, State: Initialized, Role: FOLLOWER
I20260810 18:07:14.687695 1394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251019eade1748b7a0e68e697a2a494e" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39553 } }
I20260810 18:07:14.687865 1394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:07:14.687956 1394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:07:14.688066 1394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:07:14.689255 1394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251019eade1748b7a0e68e697a2a494e" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39553 } }
I20260810 18:07:14.689625 1394 leader_election.cc:302] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 251019eade1748b7a0e68e697a2a494e; no voters:
I20260810 18:07:14.689968 1394 leader_election.cc:288] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:07:14.690099 1400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:07:14.690358 1400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [term 1 LEADER]: Becoming Leader. State: Replica: 251019eade1748b7a0e68e697a2a494e, State: Running, Role: LEADER
I20260810 18:07:14.690692 1400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251019eade1748b7a0e68e697a2a494e" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39553 } }
I20260810 18:07:14.691366 1394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:07:14.692680 1401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "251019eade1748b7a0e68e697a2a494e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251019eade1748b7a0e68e697a2a494e" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39553 } } }
I20260810 18:07:14.692694 1402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 251019eade1748b7a0e68e697a2a494e. Latest consensus state: current_term: 1 leader_uuid: "251019eade1748b7a0e68e697a2a494e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "251019eade1748b7a0e68e697a2a494e" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 39553 } } }
I20260810 18:07:14.692910 1402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [sys.catalog]: This master's current role is: LEADER
I20260810 18:07:14.693159 1401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e [sys.catalog]: This master's current role is: LEADER
I20260810 18:07:14.693334 1409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:07:14.696775 1409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:07:14.703248 1409 catalog_manager.cc:1347] Generated new cluster ID: 46694006188b486199783ea5b5659dbc
I20260810 18:07:14.703343 1409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:07:14.726552 1409 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:07:14.727599 1409 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:07:14.733666 1409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e: Generated new TSK 0
I20260810 18:07:14.734473 1409 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:07:14.775460 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39553
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:07:14.919327 1423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:07:14.919636 1423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:07:14.919682 1423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:07:14.924167 1423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:07:14.924250 1423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:07:14.924345 1423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:07:14.930045 1423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:07:14.931478 1423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:07:14.933287 1423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:07:14.945971 1428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:14.945973 1432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:14.945971 1430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:14.946221 1423 server_base.cc:1034] running on GCE node
I20260810 18:07:14.946712 1423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:07:14.947481 1423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:07:14.948679 1423 hybrid_clock.cc:648] HybridClock initialized: now 1786385234948661 us; error 40 us; skew 500 ppm
Aug 10 18:07:14 dist-test-slave-27sr krb5kdc[1299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385234, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:07:14.957368 1423 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:07:14.958652 1423 webserver.cc:460] Webserver started at http://127.21.182.65:41701/ using document root <none> and password file <none>
I20260810 18:07:14.959291 1423 fs_manager.cc:359] Metadata directory not provided
I20260810 18:07:14.959365 1423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:07:14.959546 1423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:07:14.961860 1423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "fc7c4aa8acb7432e81da77cadff7708a"
format_stamp: "Formatted at 2026-08-10 18:07:14 on dist-test-slave-27sr"
server_key: "885b6264fe954139faac3a4b42622a70"
server_key_iv: "9cc819f38b6ea0d4182c2014d0d1f453"
server_key_version: "encryptionkey@0"
I20260810 18:07:14.962476 1423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "fc7c4aa8acb7432e81da77cadff7708a"
format_stamp: "Formatted at 2026-08-10 18:07:14 on dist-test-slave-27sr"
server_key: "885b6264fe954139faac3a4b42622a70"
server_key_iv: "9cc819f38b6ea0d4182c2014d0d1f453"
server_key_version: "encryptionkey@0"
I20260810 18:07:14.967650 1423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260810 18:07:14.970749 1441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:14.972239 1423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:07:14.972385 1423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "fc7c4aa8acb7432e81da77cadff7708a"
format_stamp: "Formatted at 2026-08-10 18:07:14 on dist-test-slave-27sr"
server_key: "885b6264fe954139faac3a4b42622a70"
server_key_iv: "9cc819f38b6ea0d4182c2014d0d1f453"
server_key_version: "encryptionkey@0"
I20260810 18:07:14.972517 1423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:07:14.985960 1423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:07:14.989504 1423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:07:14.989732 1423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:07:14.990406 1423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:07:14.991355 1423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:07:14.991405 1423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:14.991500 1423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:07:14.991531 1423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:15.001464 1423 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:44589
I20260810 18:07:15.001483 1569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:44589 every 8 connection(s)
I20260810 18:07:15.002635 1423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:07:15.005657 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 1423
I20260810 18:07:15.005793 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:07:15.006114 22233 external_mini_cluster.cc:1428] Setting key a271484ed4bf6b13d08610616848005a
Aug 10 18:07:15 dist-test-slave-27sr krb5kdc[1299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385234, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
I20260810 18:07:15.018844 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.004667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:38275 (local address 127.21.182.126:39553)
0810 18:07:15.004856 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:15.004860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:15.005789 (+ 929us) server_negotiation.cc:407] Connection header received
0810 18:07:15.007516 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:15.007520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:15.007577 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:15.007657 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:15.009339 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.009893 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.011141 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.011391 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.011509 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.014628 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:07:15.014632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:07:15.014634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:07:15.014661 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:07:15.016378 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:07:15.017023 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:07:15.017029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:07:15.017036 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:07:15.017125 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:07:15.017507 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:07:15.017512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:07:15.017514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:07:15.017725 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:07:15.017860 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:15.018478 (+ 618us) server_negotiation.cc:299] Negotiation successful
0810 18:07:15.018630 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
I20260810 18:07:15.019938 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.005066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:39553 (local address 127.21.182.65:38275)
0810 18:07:15.005592 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:15.005648 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:15.007180 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:15.008124 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:15.008133 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:15.008585 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:15.009166 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.009182 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.010043 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.010048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.010976 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.010986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.011571 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.011576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.011793 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.012506 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:15.012524 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:15.014418 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:07:15.016515 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:07:15.016522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:07:15.016543 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:07:15.016838 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:07:15.017227 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:07:15.017231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:07:15.017233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:07:15.017405 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:07:15.017861 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:07:15.017869 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:07:15.018214 (+ 345us) client_negotiation.cc:769] Sending connection context
0810 18:07:15.018436 (+ 222us) client_negotiation.cc:240] Negotiation successful
0810 18:07:15.018859 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":123,"threads_started":1}
I20260810 18:07:15.021471 1570 heartbeater.cc:347] Connected to a master server at 127.21.182.126:39553
I20260810 18:07:15.021858 1570 heartbeater.cc:464] Registering TS with master...
I20260810 18:07:15.022588 1570 heartbeater.cc:511] Master 127.21.182.126:39553 requested a full tablet report, sending...
I20260810 18:07:15.024499 1353 ts_manager.cc:194] Registered new tserver with Master: fc7c4aa8acb7432e81da77cadff7708a (127.21.182.65:44589)
I20260810 18:07:15.025872 1353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:38275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:07:15.040259 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39553
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:07:15.166581 1575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:07:15.166926 1575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:07:15.166975 1575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:07:15.171535 1575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:07:15.171635 1575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:07:15.171701 1575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:07:15.177889 1575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:07:15.179255 1575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:07:15.180533 1575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:07:15.191716 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:15.191716 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:15.192181 1575 server_base.cc:1034] running on GCE node
W20260810 18:07:15.191720 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:15.192772 1575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:07:15.193500 1575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:07:15.194705 1575 hybrid_clock.cc:648] HybridClock initialized: now 1786385235194677 us; error 54 us; skew 500 ppm
Aug 10 18:07:15 dist-test-slave-27sr krb5kdc[1299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385235, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:07:15.197932 1575 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:07:15.198976 1575 webserver.cc:460] Webserver started at http://127.21.182.66:44385/ using document root <none> and password file <none>
I20260810 18:07:15.199604 1575 fs_manager.cc:359] Metadata directory not provided
I20260810 18:07:15.199667 1575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:07:15.199976 1575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:07:15.202167 1575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "a862416002574fe983e263079d3dd330"
format_stamp: "Formatted at 2026-08-10 18:07:15 on dist-test-slave-27sr"
server_key: "06af35bd44e6e9350a85a470cdfada46"
server_key_iv: "0f6a5d86e2d65a4a94bc5ea90968517b"
server_key_version: "encryptionkey@0"
I20260810 18:07:15.202764 1575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "a862416002574fe983e263079d3dd330"
format_stamp: "Formatted at 2026-08-10 18:07:15 on dist-test-slave-27sr"
server_key: "06af35bd44e6e9350a85a470cdfada46"
server_key_iv: "0f6a5d86e2d65a4a94bc5ea90968517b"
server_key_version: "encryptionkey@0"
I20260810 18:07:15.206794 1575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 18:07:15.209605 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:15.211016 1575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:07:15.211153 1575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "a862416002574fe983e263079d3dd330"
format_stamp: "Formatted at 2026-08-10 18:07:15 on dist-test-slave-27sr"
server_key: "06af35bd44e6e9350a85a470cdfada46"
server_key_iv: "0f6a5d86e2d65a4a94bc5ea90968517b"
server_key_version: "encryptionkey@0"
I20260810 18:07:15.211277 1575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:07:15.229828 1575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:07:15.233724 1575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:07:15.233976 1575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:07:15.234719 1575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:07:15.235809 1575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:07:15.235936 1575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:15.236024 1575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:07:15.236066 1575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:15.245889 1575 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:40913
I20260810 18:07:15.245915 1719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:40913 every 8 connection(s)
I20260810 18:07:15.246969 1575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:07:15.249168 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 1575
I20260810 18:07:15.249295 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:07:15.249607 22233 external_mini_cluster.cc:1428] Setting key 2c851f976eccc31f20af8e5ae7d0f06c
Aug 10 18:07:15 dist-test-slave-27sr krb5kdc[1299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385235, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
I20260810 18:07:15.264657 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.249096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:51143 (local address 127.21.182.126:39553)
0810 18:07:15.249291 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:15.249297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:15.250325 (+ 1028us) server_negotiation.cc:407] Connection header received
0810 18:07:15.252284 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:15.252288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:15.252335 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:15.252405 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:15.254073 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.254952 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.256894 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.257122 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.257255 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.260619 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:07:15.260623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:07:15.260625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:07:15.260654 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:07:15.262252 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:07:15.262855 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:07:15.262859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:07:15.262860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:07:15.262944 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:07:15.263325 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:07:15.263328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:07:15.263330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:07:15.263547 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:07:15.263657 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:15.264324 (+ 667us) server_negotiation.cc:299] Negotiation successful
0810 18:07:15.264483 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
I20260810 18:07:15.265684 1722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.249555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:39553 (local address 127.21.182.66:51143)
0810 18:07:15.250134 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:15.250183 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:15.252039 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:15.252600 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:15.252613 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:15.253197 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:15.253915 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.253932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.255136 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.255143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.256751 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.256767 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.257346 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.257353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.257604 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.258417 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:15.258448 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:15.260407 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:07:15.262390 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:07:15.262398 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:07:15.262421 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:07:15.262721 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:07:15.263045 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:07:15.263049 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:07:15.263051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:07:15.263215 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:07:15.263675 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:07:15.263683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:07:15.264051 (+ 368us) client_negotiation.cc:769] Sending connection context
0810 18:07:15.264281 (+ 230us) client_negotiation.cc:240] Negotiation successful
0810 18:07:15.264705 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":137,"threads_started":1}
I20260810 18:07:15.267068 1720 heartbeater.cc:347] Connected to a master server at 127.21.182.126:39553
I20260810 18:07:15.267409 1720 heartbeater.cc:464] Registering TS with master...
I20260810 18:07:15.268059 1720 heartbeater.cc:511] Master 127.21.182.126:39553 requested a full tablet report, sending...
I20260810 18:07:15.269361 1353 ts_manager.cc:194] Registered new tserver with Master: a862416002574fe983e263079d3dd330 (127.21.182.66:40913)
I20260810 18:07:15.269991 1353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:51143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:07:15.288090 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39553
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:07:15.407730 1725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:07:15.408092 1725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:07:15.408179 1725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:07:15.412164 1725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:07:15.412253 1725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:07:15.412348 1725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:07:15.417769 1725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:39553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:07:15.418895 1725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:07:15.419994 1725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:07:15.429988 1730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:07:15.430094 1735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:15.430073 1725 server_base.cc:1034] running on GCE node
W20260810 18:07:15.430006 1733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:07:15.430733 1725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:07:15.431427 1725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:07:15.432621 1725 hybrid_clock.cc:648] HybridClock initialized: now 1786385235432583 us; error 48 us; skew 500 ppm
Aug 10 18:07:15 dist-test-slave-27sr krb5kdc[1299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385235, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:07:15.435756 1725 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:07:15.436803 1725 webserver.cc:460] Webserver started at http://127.21.182.67:42195/ using document root <none> and password file <none>
I20260810 18:07:15.437379 1725 fs_manager.cc:359] Metadata directory not provided
I20260810 18:07:15.437445 1725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:07:15.437623 1725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:07:15.439771 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "de5727c559b84f588b5bfb2211561112"
format_stamp: "Formatted at 2026-08-10 18:07:15 on dist-test-slave-27sr"
server_key: "129f3a4b3a26b75ce4a4a18b5b87f5bb"
server_key_iv: "37d430e3040b2190f45f8c9969b8de94"
server_key_version: "encryptionkey@0"
I20260810 18:07:15.440420 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "de5727c559b84f588b5bfb2211561112"
format_stamp: "Formatted at 2026-08-10 18:07:15 on dist-test-slave-27sr"
server_key: "129f3a4b3a26b75ce4a4a18b5b87f5bb"
server_key_iv: "37d430e3040b2190f45f8c9969b8de94"
server_key_version: "encryptionkey@0"
I20260810 18:07:15.444473 1725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:07:15.447090 1742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:15.448470 1725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:07:15.448575 1725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "de5727c559b84f588b5bfb2211561112"
format_stamp: "Formatted at 2026-08-10 18:07:15 on dist-test-slave-27sr"
server_key: "129f3a4b3a26b75ce4a4a18b5b87f5bb"
server_key_iv: "37d430e3040b2190f45f8c9969b8de94"
server_key_version: "encryptionkey@0"
I20260810 18:07:15.448662 1725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:07:15.487296 1725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:07:15.490984 1725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:07:15.491163 1725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:07:15.491927 1725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:07:15.492938 1725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:07:15.493022 1725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:15.493119 1725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:07:15.493170 1725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:07:15.502741 1725 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:44653
I20260810 18:07:15.502765 1871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:44653 every 8 connection(s)
I20260810 18:07:15.503779 1725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:07:15.506896 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 1725
I20260810 18:07:15.507023 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:07:15.507319 22233 external_mini_cluster.cc:1428] Setting key 38b51061100c9d76ce8e8ba171addf91
Aug 10 18:07:15 dist-test-slave-27sr krb5kdc[1299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385235, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:07:15.518312 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.505773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:47101 (local address 127.21.182.126:39553)
0810 18:07:15.505990 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:15.505996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:15.506833 (+ 837us) server_negotiation.cc:407] Connection header received
0810 18:07:15.508595 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:15.508598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:15.508656 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:15.508737 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:15.509929 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.510480 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.511621 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.511852 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.512004 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.514570 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:07:15.514580 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:07:15.514582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:07:15.514611 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:07:15.516194 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:07:15.516758 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:07:15.516766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:07:15.516768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:07:15.516848 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:07:15.517196 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:07:15.517200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:07:15.517202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:07:15.517394 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:07:15.517507 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:15.518015 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 18:07:15.518143 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 18:07:15.519186 1874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.506180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:39553 (local address 127.21.182.67:47101)
0810 18:07:15.506670 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:15.506712 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:15.508363 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:15.508917 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:15.508924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:15.509291 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:07:15.509772 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.509785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.510611 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.510617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.511491 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.511500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.512044 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.512048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.512220 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.512773 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:07:15.512789 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:07:15.514368 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:07:15.516331 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:07:15.516338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:07:15.516356 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:07:15.516627 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:07:15.516945 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:07:15.516950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:07:15.516952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:07:15.517093 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:07:15.517507 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:07:15.517514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:07:15.517774 (+ 260us) client_negotiation.cc:769] Sending connection context
0810 18:07:15.517973 (+ 199us) client_negotiation.cc:240] Negotiation successful
0810 18:07:15.518327 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":137,"threads_started":1}
I20260810 18:07:15.520597 1872 heartbeater.cc:347] Connected to a master server at 127.21.182.126:39553
I20260810 18:07:15.520865 1872 heartbeater.cc:464] Registering TS with master...
I20260810 18:07:15.521400 1872 heartbeater.cc:511] Master 127.21.182.126:39553 requested a full tablet report, sending...
I20260810 18:07:15.522533 1353 ts_manager.cc:194] Registered new tserver with Master: de5727c559b84f588b5bfb2211561112 (127.21.182.67:44653)
I20260810 18:07:15.523124 1353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:47101
I20260810 18:07:15.533242 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:07:15.543081 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.535310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.21.182.126:39553)
0810 18:07:15.535460 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:15.535464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:15.535660 (+ 196us) server_negotiation.cc:407] Connection header received
0810 18:07:15.535755 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:15.535757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:15.535797 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:15.535948 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:07:15.536778 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.537291 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.538194 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.538404 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.538481 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.540025 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:07:15.540028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:07:15.540030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:07:15.540054 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:07:15.541344 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:07:15.541826 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:07:15.541829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:07:15.541830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:07:15.541929 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:07:15.542188 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:07:15.542190 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:07:15.542191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:07:15.542405 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:07:15.542506 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:15.542830 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 18:07:15.542957 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 18:07:15.546521 1353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:07:15.549990 1353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:07:15.565959 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.559100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:44653 (local address 127.0.0.1:60988)
0810 18:07:15.559509 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:15.559565 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:15.559702 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:15.561060 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:15.561069 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:15.561100 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:07:15.561294 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.561301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.562831 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.562834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.564346 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.564358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.565220 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.565224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.565450 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.565563 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 18:07:15.565633 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 18:07:15.565791 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":289,"thread_start_us":93,"threads_started":1}
I20260810 18:07:15.566097 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.560357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.21.182.65:44589)
0810 18:07:15.560685 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:15.560692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:15.560845 (+ 153us) server_negotiation.cc:407] Connection header received
0810 18:07:15.560962 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:15.560977 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:15.561157 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:15.561328 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:07:15.561562 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.562371 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.564574 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.565376 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.565479 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.565619 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:15.565822 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 18:07:15.565936 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":128,"threads_started":1}
I20260810 18:07:15.565972 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.559481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:40913 (local address 127.0.0.1:39418)
0810 18:07:15.560033 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:15.560046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:15.560134 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:15.561137 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:15.561140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:15.561162 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:07:15.561294 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.561301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.562832 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.562834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.564169 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.564180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.565253 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.565257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.565456 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.565523 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 18:07:15.565625 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 18:07:15.565771 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":114,"spinlock_wait_cycles":1280,"thread_start_us":115,"threads_started":1}
I20260810 18:07:15.566685 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.559247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.21.182.67:44653)
0810 18:07:15.560440 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:15.560458 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:15.560489 (+ 31us) server_negotiation.cc:407] Connection header received
0810 18:07:15.560591 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:15.560596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:15.560841 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:15.561047 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:07:15.561435 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.562662 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.564495 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.565081 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.565217 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.565386 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:15.566450 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0810 18:07:15.566565 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1073,"thread_start_us":182,"threads_started":1}
I20260810 18:07:15.567021 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.559923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:44589 (local address 127.0.0.1:43794)
0810 18:07:15.560732 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:15.560746 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:15.560854 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:15.561323 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:15.561326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:15.561344 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:07:15.561450 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.561456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.562713 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.562719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.564453 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.564463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.565495 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.565499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.565646 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.565666 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:07:15.565727 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 18:07:15.566897 (+ 1170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":1465,"thread_start_us":87,"threads_started":1}
I20260810 18:07:15.567030 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.560031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39418 (local address 127.21.182.66:40913)
0810 18:07:15.560606 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:15.560625 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:15.560651 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:07:15.560747 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:15.560755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:15.560968 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:15.561462 (+ 494us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:07:15.561620 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.562700 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.564293 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.565102 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.566071 (+ 969us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.566708 (+ 637us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:15.566783 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 18:07:15.566888 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":146,"threads_started":1}
I20260810 18:07:15.569550 1794 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9a1f413760458ebe50c74861fb9ab3 (DEFAULT_TABLE table=test-table [id=41086a2dadc848a385f78f6a3b5b05d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:07:15.569737 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9a1f413760458ebe50c74861fb9ab3 (DEFAULT_TABLE table=test-table [id=41086a2dadc848a385f78f6a3b5b05d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:07:15.569914 1494 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9a1f413760458ebe50c74861fb9ab3 (DEFAULT_TABLE table=test-table [id=41086a2dadc848a385f78f6a3b5b05d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:07:15.570708 1794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9a1f413760458ebe50c74861fb9ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:07:15.570801 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9a1f413760458ebe50c74861fb9ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:07:15.570811 1494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9a1f413760458ebe50c74861fb9ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:07:15.578253 1891 tablet_bootstrap.cc:492] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112: Bootstrap starting.
I20260810 18:07:15.578896 1893 tablet_bootstrap.cc:492] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330: Bootstrap starting.
I20260810 18:07:15.580250 1892 tablet_bootstrap.cc:492] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a: Bootstrap starting.
I20260810 18:07:15.581175 1893 tablet_bootstrap.cc:654] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330: Neither blocks nor log segments found. Creating new log.
I20260810 18:07:15.581326 1891 tablet_bootstrap.cc:654] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112: Neither blocks nor log segments found. Creating new log.
I20260810 18:07:15.582029 1893 log.cc:824] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330: Log is configured to *not* fsync() on all Append() calls
I20260810 18:07:15.582154 1891 log.cc:824] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112: Log is configured to *not* fsync() on all Append() calls
I20260810 18:07:15.582491 1892 tablet_bootstrap.cc:654] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a: Neither blocks nor log segments found. Creating new log.
I20260810 18:07:15.583298 1892 log.cc:824] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:07:15.584470 1893 tablet_bootstrap.cc:492] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330: No bootstrap required, opened a new log
I20260810 18:07:15.584502 1891 tablet_bootstrap.cc:492] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112: No bootstrap required, opened a new log
I20260810 18:07:15.584692 1893 ts_tablet_manager.cc:1397] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:07:15.584692 1891 ts_tablet_manager.cc:1397] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:07:15.585422 1892 tablet_bootstrap.cc:492] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a: No bootstrap required, opened a new log
I20260810 18:07:15.585633 1892 ts_tablet_manager.cc:1397] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:07:15.587724 1893 raft_consensus.cc:348] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5727c559b84f588b5bfb2211561112" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 44653 } } peers { permanent_uuid: "fc7c4aa8acb7432e81da77cadff7708a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44589 } } peers { permanent_uuid: "a862416002574fe983e263079d3dd330" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40913 } }
I20260810 18:07:15.588073 1893 raft_consensus.cc:374] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:07:15.588129 1893 raft_consensus.cc:729] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a862416002574fe983e263079d3dd330, State: Initialized, Role: FOLLOWER
I20260810 18:07:15.588611 1893 consensus_queue.cc:260] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5727c559b84f588b5bfb2211561112" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 44653 } } peers { permanent_uuid: "fc7c4aa8acb7432e81da77cadff7708a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44589 } } peers { permanent_uuid: "a862416002574fe983e263079d3dd330" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40913 } }
I20260810 18:07:15.588740 1891 raft_consensus.cc:348] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5727c559b84f588b5bfb2211561112" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 44653 } } peers { permanent_uuid: "fc7c4aa8acb7432e81da77cadff7708a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44589 } } peers { permanent_uuid: "a862416002574fe983e263079d3dd330" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40913 } }
I20260810 18:07:15.589007 1891 raft_consensus.cc:374] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:07:15.589066 1891 raft_consensus.cc:729] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5727c559b84f588b5bfb2211561112, State: Initialized, Role: FOLLOWER
I20260810 18:07:15.589464 1720 heartbeater.cc:503] Master 127.21.182.126:39553 was elected leader, sending a full tablet report...
I20260810 18:07:15.589529 1891 consensus_queue.cc:260] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5727c559b84f588b5bfb2211561112" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 44653 } } peers { permanent_uuid: "fc7c4aa8acb7432e81da77cadff7708a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44589 } } peers { permanent_uuid: "a862416002574fe983e263079d3dd330" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40913 } }
I20260810 18:07:15.589994 1892 raft_consensus.cc:348] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5727c559b84f588b5bfb2211561112" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 44653 } } peers { permanent_uuid: "fc7c4aa8acb7432e81da77cadff7708a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44589 } } peers { permanent_uuid: "a862416002574fe983e263079d3dd330" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40913 } }
I20260810 18:07:15.590240 1892 raft_consensus.cc:374] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:07:15.590286 1892 raft_consensus.cc:729] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7c4aa8acb7432e81da77cadff7708a, State: Initialized, Role: FOLLOWER
I20260810 18:07:15.590546 1891 ts_tablet_manager.cc:1428] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:07:15.590708 1892 consensus_queue.cc:260] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5727c559b84f588b5bfb2211561112" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 44653 } } peers { permanent_uuid: "fc7c4aa8acb7432e81da77cadff7708a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44589 } } peers { permanent_uuid: "a862416002574fe983e263079d3dd330" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40913 } }
I20260810 18:07:15.590915 1872 heartbeater.cc:503] Master 127.21.182.126:39553 was elected leader, sending a full tablet report...
I20260810 18:07:15.591722 1892 ts_tablet_manager.cc:1428] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:07:15.592979 1570 heartbeater.cc:503] Master 127.21.182.126:39553 was elected leader, sending a full tablet report...
I20260810 18:07:15.594153 1893 ts_tablet_manager.cc:1428] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
W20260810 18:07:15.748317 1721 tablet.cc:2369] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:07:15.754020 1571 tablet.cc:2369] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:07:15.754819 1873 tablet.cc:2369] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:07:15.792935 1898 raft_consensus.cc:482] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:07:15.793187 1898 raft_consensus.cc:504] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5727c559b84f588b5bfb2211561112" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 44653 } } peers { permanent_uuid: "fc7c4aa8acb7432e81da77cadff7708a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44589 } } peers { permanent_uuid: "a862416002574fe983e263079d3dd330" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40913 } }
I20260810 18:07:15.794660 1898 leader_election.cc:288] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc7c4aa8acb7432e81da77cadff7708a (127.21.182.65:44589), a862416002574fe983e263079d3dd330 (127.21.182.66:40913)
I20260810 18:07:15.799108 1874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.794844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:40913 (local address 127.21.182.67:44145)
0810 18:07:15.794955 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:15.794973 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:15.795085 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:15.795455 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:15.795458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:15.795480 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:07:15.795613 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.795619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.796682 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.796685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.797881 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.797892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.798704 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.798706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.798789 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.798817 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:07:15.798886 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 18:07:15.798944 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 18:07:15.799154 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.795013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:44145 (local address 127.21.182.66:40913)
0810 18:07:15.795175 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:15.795180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:15.795196 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:07:15.795253 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:15.795259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:15.795322 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:15.795419 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:07:15.795797 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.796576 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.798024 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.798576 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.798716 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.798773 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:15.798932 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 18:07:15.799015 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:07:15.799485 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.795089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:56803 (local address 127.21.182.65:44589)
0810 18:07:15.795238 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:15.795242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:15.795767 (+ 525us) server_negotiation.cc:407] Connection header received
0810 18:07:15.795858 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:15.795862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:15.795942 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:15.796021 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:07:15.796273 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.796910 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.798236 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.798897 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.799015 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.799060 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:15.799277 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 18:07:15.799395 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:07:15.799866 1901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.794854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:44589 (local address 127.21.182.67:56803)
0810 18:07:15.795541 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:15.795555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:07:15.795770 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:07:15.796042 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:07:15.796044 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:07:15.796059 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:07:15.796162 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.796166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.797026 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.797029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.798129 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:07:15.798137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.799020 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.799023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:07:15.799108 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.799140 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 18:07:15.799649 (+ 509us) client_negotiation.cc:240] Negotiation successful
0810 18:07:15.799740 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":310,"thread_start_us":139,"threads_started":1}
I20260810 18:07:15.799865 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9a1f413760458ebe50c74861fb9ab3" candidate_uuid: "de5727c559b84f588b5bfb2211561112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a862416002574fe983e263079d3dd330" is_pre_election: true
I20260810 18:07:15.800244 1667 raft_consensus.cc:2454] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de5727c559b84f588b5bfb2211561112 in term 0.
I20260810 18:07:15.800750 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9a1f413760458ebe50c74861fb9ab3" candidate_uuid: "de5727c559b84f588b5bfb2211561112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7c4aa8acb7432e81da77cadff7708a" is_pre_election: true
I20260810 18:07:15.800781 1746 leader_election.cc:302] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a862416002574fe983e263079d3dd330, de5727c559b84f588b5bfb2211561112; no voters:
I20260810 18:07:15.801098 1516 raft_consensus.cc:2454] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de5727c559b84f588b5bfb2211561112 in term 0.
I20260810 18:07:15.801126 1898 raft_consensus.cc:2781] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:07:15.801254 1898 raft_consensus.cc:482] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:07:15.801335 1898 raft_consensus.cc:3037] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:07:15.802559 1898 raft_consensus.cc:504] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5727c559b84f588b5bfb2211561112" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 44653 } } peers { permanent_uuid: "fc7c4aa8acb7432e81da77cadff7708a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44589 } } peers { permanent_uuid: "a862416002574fe983e263079d3dd330" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40913 } }
I20260810 18:07:15.802910 1898 leader_election.cc:288] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [CANDIDATE]: Term 1 election: Requested vote from peers fc7c4aa8acb7432e81da77cadff7708a (127.21.182.65:44589), a862416002574fe983e263079d3dd330 (127.21.182.66:40913)
I20260810 18:07:15.803330 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9a1f413760458ebe50c74861fb9ab3" candidate_uuid: "de5727c559b84f588b5bfb2211561112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7c4aa8acb7432e81da77cadff7708a"
I20260810 18:07:15.803362 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9a1f413760458ebe50c74861fb9ab3" candidate_uuid: "de5727c559b84f588b5bfb2211561112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a862416002574fe983e263079d3dd330"
I20260810 18:07:15.803472 1516 raft_consensus.cc:3037] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:07:15.803472 1667 raft_consensus.cc:3037] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:07:15.804854 1516 raft_consensus.cc:2454] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de5727c559b84f588b5bfb2211561112 in term 1.
I20260810 18:07:15.804850 1667 raft_consensus.cc:2454] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de5727c559b84f588b5bfb2211561112 in term 1.
I20260810 18:07:15.805155 1746 leader_election.cc:302] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a862416002574fe983e263079d3dd330, de5727c559b84f588b5bfb2211561112; no voters:
I20260810 18:07:15.805431 1898 raft_consensus.cc:2781] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:07:15.805671 1898 raft_consensus.cc:686] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [term 1 LEADER]: Becoming Leader. State: Replica: de5727c559b84f588b5bfb2211561112, State: Running, Role: LEADER
I20260810 18:07:15.806057 1898 consensus_queue.cc:237] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5727c559b84f588b5bfb2211561112" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 44653 } } peers { permanent_uuid: "fc7c4aa8acb7432e81da77cadff7708a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44589 } } peers { permanent_uuid: "a862416002574fe983e263079d3dd330" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40913 } }
I20260810 18:07:15.809324 1351 catalog_manager.cc:5495] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 reported cstate change: term changed from 0 to 1, leader changed from <none> to de5727c559b84f588b5bfb2211561112 (127.21.182.67). New cstate: current_term: 1 leader_uuid: "de5727c559b84f588b5bfb2211561112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de5727c559b84f588b5bfb2211561112" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 44653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc7c4aa8acb7432e81da77cadff7708a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a862416002574fe983e263079d3dd330" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40913 } health_report { overall_health: UNKNOWN } } }
I20260810 18:07:15.860529 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.857014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.21.182.67:44653)
0810 18:07:15.857149 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:07:15.857153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:07:15.857168 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:07:15.857346 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:07:15.857349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:07:15.857412 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:07:15.857507 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:07:15.857746 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.858255 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.859076 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:07:15.859286 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:07:15.859365 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:07:15.859664 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:07:15.860172 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:07:15.860266 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:07:15.860332 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 18:07:15.860387 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:07:15.866789 1667 raft_consensus.cc:1261] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330 [term 1 FOLLOWER]: Refusing update from remote peer de5727c559b84f588b5bfb2211561112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:07:15.866789 1516 raft_consensus.cc:1261] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a [term 1 FOLLOWER]: Refusing update from remote peer de5727c559b84f588b5bfb2211561112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:07:15.867584 1902 consensus_queue.cc:1035] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc7c4aa8acb7432e81da77cadff7708a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:07:15.867810 1898 consensus_queue.cc:1035] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a862416002574fe983e263079d3dd330" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:07:15.876024 1906 mvcc.cc:205] Tried to move back new op lower bound from 7317033926102564864 to 7317033925864128512. Current Snapshot: MvccSnapshot[applied={T|T < 7317033926102564864}]
I20260810 18:07:15.877022 1907 mvcc.cc:205] Tried to move back new op lower bound from 7317033926102564864 to 7317033925864128512. Current Snapshot: MvccSnapshot[applied={T|T < 7317033926102564864}]
I20260810 18:07:15.886984 1353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:07:15.887174 1353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:07:15.890118 1353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 251019eade1748b7a0e68e697a2a494e: Sending DeleteTablet for 3 replicas of tablet 0c9a1f413760458ebe50c74861fb9ab3
I20260810 18:07:15.891222 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9a1f413760458ebe50c74861fb9ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:07:15 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:39418
I20260810 18:07:15.891347 1794 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9a1f413760458ebe50c74861fb9ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:07:15 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:60988
I20260810 18:07:15.891379 1494 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9a1f413760458ebe50c74861fb9ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:07:15 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:43794
I20260810 18:07:15.891949 1916 tablet_replica.cc:330] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a: stopping tablet replica
I20260810 18:07:15.892149 1915 tablet_replica.cc:330] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330: stopping tablet replica
I20260810 18:07:15.892333 1916 raft_consensus.cc:2229] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:07:15.892558 1915 raft_consensus.cc:2229] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:07:15.892717 1916 raft_consensus.cc:2258] T 0c9a1f413760458ebe50c74861fb9ab3 P fc7c4aa8acb7432e81da77cadff7708a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:07:15.892858 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 1423
I20260810 18:07:15.892901 1915 raft_consensus.cc:2258] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:07:15.893323 1917 tablet_replica.cc:330] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112: stopping tablet replica
I20260810 18:07:15.893682 1917 raft_consensus.cc:2229] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:07:15.894115 1917 raft_consensus.cc:2258] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:07:15.895699 1915 ts_tablet_manager.cc:1905] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:07:15.896169 1917 ts_tablet_manager.cc:1905] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:07:15.899070 1917 ts_tablet_manager.cc:1918] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:07:15.899168 1917 log.cc:1196] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/0c9a1f413760458ebe50c74861fb9ab3
I20260810 18:07:15.899354 1915 ts_tablet_manager.cc:1918] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:07:15.899477 1915 log.cc:1196] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/0c9a1f413760458ebe50c74861fb9ab3
I20260810 18:07:15.899533 1917 ts_tablet_manager.cc:1939] T 0c9a1f413760458ebe50c74861fb9ab3 P de5727c559b84f588b5bfb2211561112: Deleting consensus metadata
I20260810 18:07:15.899832 1915 ts_tablet_manager.cc:1939] T 0c9a1f413760458ebe50c74861fb9ab3 P a862416002574fe983e263079d3dd330: Deleting consensus metadata
I20260810 18:07:15.900995 1337 catalog_manager.cc:4841] TS de5727c559b84f588b5bfb2211561112 (127.21.182.67:44653): tablet 0c9a1f413760458ebe50c74861fb9ab3 (table test-table [id=41086a2dadc848a385f78f6a3b5b05d9]) successfully deleted
I20260810 18:07:15.901193 1336 catalog_manager.cc:4841] TS a862416002574fe983e263079d3dd330 (127.21.182.66:40913): tablet 0c9a1f413760458ebe50c74861fb9ab3 (table test-table [id=41086a2dadc848a385f78f6a3b5b05d9]) successfully deleted
W20260810 18:07:15.906438 1337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:44589 (error 108)
I20260810 18:07:15.907457 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 1575
I20260810 18:07:15.907750 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:07:15.907370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:44589 (local address 127.0.0.1:43802)
0810 18:07:15.907546 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 18:07:15.907634 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:44589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 18:07:15.911475 1337 catalog_manager.cc:4584] TS fc7c4aa8acb7432e81da77cadff7708a (127.21.182.65:44589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c9a1f413760458ebe50c74861fb9ab3: Network error: Client connection negotiation failed: client connection to 127.21.182.65:44589: connect: Connection refused (error 111)
I20260810 18:07:15.920543 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 1725
I20260810 18:07:15.930516 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 1314
2026-08-10T18:07:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2793 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:07:15 dist-test-slave-27sr krb5kdc[1921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:07:15 dist-test-slave-27sr krb5kdc[1921](info): set up 2 sockets
Aug 10 18:07:15 dist-test-slave-27sr krb5kdc[1921](info): commencing operation
krb5kdc: starting...
W20260810 18:07:17.051405 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:07:17 dist-test-slave-27sr krb5kdc[1921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:07:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:07:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/rangeradmin_127.21.182.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/rangeradmin_127.21.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/rangerlookup_127.21.182.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/rangerlookup_127.21.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/HTTP_127.21.182.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/HTTP_127.21.182.84@KRBTEST.COM.keytab.
I20260810 18:07:17.246541 22233 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0GlVnd/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/postgres -l logfile start
2026-08-10 18:07:20.068 UTC [1952] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 18:07:20.068 UTC [1952] LOG: listening on IPv4 address "127.21.182.84", port 52231
2026-08-10 18:07:20.072 UTC [1952] LOG: listening on Unix socket "/tmp/.s.PGSQL.52231"
2026-08-10 18:07:20.096 UTC [1955] LOG: database system was shut down at 2026-08-10 18:07:18 UTC
2026-08-10 18:07:20.100 UTC [1952] LOG: database system is ready to accept connections
I20260810 18:07:21.157935 22233 mini_postgres.cc:96] Postgres bound to 52231
2026-08-10 18:07:21.167 UTC [1963] FATAL: database "slave" does not exist
127.21.182.84:52231 - accepting connections
I20260810 18:07:21.168372 22233 mini_ranger.cc:161] Starting Ranger...
I20260810 18:07:21.186447 22233 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 18:07:21.755578 22233 mini_ranger.cc:87] Created ranger Postgres database
I20260810 18:07:21.755813 22233 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-admin
2026-08-10 18:07:22,091 [I] DB FLAVOR :POSTGRES
2026-08-10 18:07:22,092 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 18:07:22,092 [I] Checking connection..
2026-08-10 18:07:22,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 18:07:22,346 [I] Checking connection passed.
2026-08-10 18:07:22,346 [I] --------- Verifying version history table ---------
2026-08-10 18:07:22,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:07:22,656 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 18:07:22,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:07:22,910 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 18:07:22,910 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 18:07:22,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 18:07:23.166 UTC [2034] WARNING: there is no transaction in progress
2026-08-10 18:07:23,175 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 18:07:23,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:07:23,421 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 18:07:23,422 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 18:07:23,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 18:07:23,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 18:07:23,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-27sr.c.gcp-upstream.internal','N') ;"
2026-08-10 18:07:24,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 18:07:24,440 [I] Table x_portal_user does not exist in database ranger
2026-08-10 18:07:24,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 18:07:24,711 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 18:07:24,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 18:07:24,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 18:07:25,211 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 18:07:25,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 18:07:26.321 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.341 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.360 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.379 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.396 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.418 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.482 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.490 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.508 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.520 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.531 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.548 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.559 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.566 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.573 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:26.581 UTC [2151] WARNING: there is no transaction in progress
2026-08-10 18:07:27,340 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 18:07:27,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 18:07:27,591 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 18:07:27,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 18:07:27,851 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 18:07:27,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 18:07:28,131 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 18:07:28,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 18:07:28,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 18:07:28,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-27sr.c.gcp-upstream.internal';"
2026-08-10 18:07:28,899 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 18:07:28,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 18:07:29,159 [I] DB_PATCHES have already been applied
I20260810 18:07:29.166165 22233 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-admin:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0GlVnd/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 18:07:29.166249 22233 mini_ranger.cc:193] Using host: 127.21.182.84
I20260810 18:07:29.171469 22233 mini_ranger.cc:227] Ranger admin URL: http://127.21.182.84:52823
Aug 10, 2026 6:07:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:07:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:07:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 6:07:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 6:07:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 6:07:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 6:07:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 18:07:30.442281 22233 mini_ranger.cc:160] Time spent starting Ranger: real 9.274s user 0.002s sys 0.013s
Aug 10, 2026 6:07:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 6:07:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 6:07:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 6:07:51 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 6:07:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 6:07:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 6:07:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 6:07:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 6:07:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 6:07:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 6:07:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 6:07:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 6:07:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 6:07:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 6:07:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 6:07:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 6:07:51 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 6:07:52 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 18:07:52.732400 22233 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/rangerkms_127.21.182.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/rangerkms_127.21.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.182.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.182.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.182.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/HTTP_127.21.182.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.182.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/HTTP_127.21.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 18:07:52 dist-test-slave-27sr krb5kdc[1921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385272, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 18:07:52.834872 22233 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 18:07:52.846747 22233 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 18:07:53.831856 22233 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 18:07:53.832000 22233 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-kms
I20260810 18:07:53.832119 22233 mini_ranger_kms.cc:227] Using postgres at 127.21.182.84:52231
2026-08-10 18:07:54,559 [I] DB FLAVOR :POSTGRES
2026-08-10 18:07:54,560 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 18:07:54,560 [I] Checking connection
2026-08-10 18:07:54,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 18:07:54,855 [I] connection success
2026-08-10 18:07:54,856 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 18:07:54,856 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 18:07:54,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 18:07:55,122 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 18:07:55,122 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 18:07:55,122 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 18:07:55,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:52231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 18:07:55,394 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 18:07:55.884788 22233 mini_ranger_kms.cc:310] Created kms service
I20260810 18:07:56.121757 22233 mini_ranger_kms.cc:326] Created kudu user
I20260810 18:07:56.172513 22233 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 18:07:56 dist-test-slave-27sr krb5kdc[1921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385272, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:07:56.646261 22233 mini_ranger_kms.cc:384] Added ranger policy
I20260810 18:07:56.646457 22233 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-kms:/tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 18:07:56.646486 22233 mini_ranger_kms.cc:240] Using host: 127.21.182.84
I20260810 18:07:56.652302 22233 mini_ranger_kms.cc:276] Ranger KMS PID: 2356
I20260810 18:07:56.652570 22233 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.182.84:50209
Aug 10, 2026 6:07:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:07:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:07:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 6:07:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 6:07:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 6:07:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 18:07:59 dist-test-slave-27sr krb5kdc[1921](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786385279, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.182.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:07:59.228787 22233 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.394s user 0.048s sys 0.428s
I20260810 18:07:59.228951 22233 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 18:07:59.229107 22233 mini_ranger_kms.cc:401] 127.21.182.84:50209/kms/v1/keys
Aug 10 18:07:59 dist-test-slave-27sr krb5kdc[1921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385279, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.182.84@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
Aug 10 18:08:00 dist-test-slave-27sr krb5kdc[1921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385279, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.182.84@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
Aug 10 18:08:02 dist-test-slave-27sr krb5kdc[1921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:08:02.752671 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:33355
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:33355
--ranger_config_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.182.84:50209/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:08:02.893832 2412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:08:02.894140 2412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:08:02.894182 2412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:08:02.898342 2412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:08:02.898418 2412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:08:02.898438 2412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:08:02.898455 2412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:08:02.898470 2412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:08:02.904529 2412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.182.84:50209/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:33355
--ranger_config_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:33355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:08:02.905711 2412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:08:02.906827 2412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:08:02.916756 2419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:02.916716 2418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:02.916942 2421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:02.917061 2412 server_base.cc:1034] running on GCE node
I20260810 18:08:02.918026 2412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:08:02.919651 2412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:08:02.920856 2412 hybrid_clock.cc:648] HybridClock initialized: now 1786385282920828 us; error 45 us; skew 500 ppm
Aug 10 18:08:02 dist-test-slave-27sr krb5kdc[1921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385282, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:08:02.924966 2412 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:08:02.926316 2412 webserver.cc:460] Webserver started at http://127.21.182.126:38275/ using document root <none> and password file <none>
I20260810 18:08:02.927001 2412 fs_manager.cc:359] Metadata directory not provided
I20260810 18:08:02.927069 2412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:08:02.927331 2412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:08:02 dist-test-slave-27sr krb5kdc[1921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385282, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:02.993848 2412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "08f19d2592e04c2fb8e6cad2352671f0"
format_stamp: "Formatted at 2026-08-10 18:08:02 on dist-test-slave-27sr"
server_key: "946f152bc535b6e046afe67b1ebfccb1"
server_key_iv: "95045973625e3586862cc6d3429c292e"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:02.994730 2412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "08f19d2592e04c2fb8e6cad2352671f0"
format_stamp: "Formatted at 2026-08-10 18:08:02 on dist-test-slave-27sr"
server_key: "946f152bc535b6e046afe67b1ebfccb1"
server_key_iv: "95045973625e3586862cc6d3429c292e"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:03.000450 2412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 18:08:03.019985 2429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:03.021795 2412 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:08:03.021950 2412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "08f19d2592e04c2fb8e6cad2352671f0"
format_stamp: "Formatted at 2026-08-10 18:08:02 on dist-test-slave-27sr"
server_key: "946f152bc535b6e046afe67b1ebfccb1"
server_key_iv: "95045973625e3586862cc6d3429c292e"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:03.022102 2412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:08:03.056450 2412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:08:03.060329 2412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:08:03.060551 2412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:08:03.070766 2412 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:33355
I20260810 18:08:03.071014 2492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:33355 every 8 connection(s)
I20260810 18:08:03.072144 2412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:08:03.073479 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 2412
I20260810 18:08:03.073601 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:08:03.073902 22233 external_mini_cluster.cc:1428] Setting key be453f01ef1f9cca6c85cc513495e69b
I20260810 18:08:03.076076 2493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:08:03.084192 2493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0: Bootstrap starting.
Aug 10 18:08:03 dist-test-slave-27sr krb5kdc[1921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:08:03.087607 2493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0: Neither blocks nor log segments found. Creating new log.
I20260810 18:08:03.089198 2493 log.cc:824] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0: Log is configured to *not* fsync() on all Append() calls
I20260810 18:08:03.092344 2493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0: No bootstrap required, opened a new log
I20260810 18:08:03.093371 2496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:03.075449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.21.182.126:33355)
0810 18:08:03.076213 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:03.076261 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:03.076306 (+ 45us) server_negotiation.cc:407] Connection header received
0810 18:08:03.077247 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:03.077271 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:03.077723 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:03.078121 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:08:03.079148 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:03.080865 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:03.082065 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:03.082877 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:03.083057 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:03.086799 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:08:03.086806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:08:03.086826 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:08:03.086868 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:08:03.089495 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:03.090236 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:03.090241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:03.090247 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:03.090386 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:03.090761 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:03.090764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:03.090766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:03.091357 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:08:03.091562 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:03.091962 (+ 400us) server_negotiation.cc:299] Negotiation successful
0810 18:08:03.092298 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":228,"threads_started":1}
I20260810 18:08:03.096457 2493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f19d2592e04c2fb8e6cad2352671f0" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33355 } }
I20260810 18:08:03.096707 2493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:08:03.096788 2493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f19d2592e04c2fb8e6cad2352671f0, State: Initialized, Role: FOLLOWER
I20260810 18:08:03.097234 2493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f19d2592e04c2fb8e6cad2352671f0" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33355 } }
I20260810 18:08:03.097366 2493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:08:03.097461 2493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:08:03.097573 2493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:08:03.098779 2493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f19d2592e04c2fb8e6cad2352671f0" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33355 } }
I20260810 18:08:03.099188 2493 leader_election.cc:302] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08f19d2592e04c2fb8e6cad2352671f0; no voters:
I20260810 18:08:03.099583 2493 leader_election.cc:288] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:08:03.099792 2498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:08:03.100190 2498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [term 1 LEADER]: Becoming Leader. State: Replica: 08f19d2592e04c2fb8e6cad2352671f0, State: Running, Role: LEADER
I20260810 18:08:03.100651 2498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f19d2592e04c2fb8e6cad2352671f0" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33355 } }
I20260810 18:08:03.101146 2493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:08:03.102221 2498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08f19d2592e04c2fb8e6cad2352671f0. Latest consensus state: current_term: 1 leader_uuid: "08f19d2592e04c2fb8e6cad2352671f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f19d2592e04c2fb8e6cad2352671f0" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33355 } } }
I20260810 18:08:03.102344 2498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [sys.catalog]: This master's current role is: LEADER
I20260810 18:08:03.103510 2500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08f19d2592e04c2fb8e6cad2352671f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f19d2592e04c2fb8e6cad2352671f0" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33355 } } }
I20260810 18:08:03.103730 2500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0 [sys.catalog]: This master's current role is: LEADER
I20260810 18:08:03.104287 2505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:08:03.107935 2493 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 18:08:03.108458 2505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:08:03.124150 2505 catalog_manager.cc:1347] Generated new cluster ID: 890d80e8e3fa4b1a98bb9229f09872ee
I20260810 18:08:03.124256 2505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:08:03.134097 2505 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:08:03.135489 2505 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:08:03.147010 2505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0: Generated new TSK 0
I20260810 18:08:03.148074 2505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:08:03.971588 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.182.84:50209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33355
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:08:04.145597 2540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:08:04.146100 2540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:08:04.146245 2540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:08:04.152710 2540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:08:04.152807 2540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:08:04.152940 2540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:08:04.159283 2540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.182.84:50209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:08:04.160714 2540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:08:04.162241 2540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:08:04.174929 2547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:04.174993 2548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:04.175356 2540 server_base.cc:1034] running on GCE node
W20260810 18:08:04.175670 2550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:04.176414 2540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:08:04.177218 2540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:08:04.178401 2540 hybrid_clock.cc:648] HybridClock initialized: now 1786385284178385 us; error 46 us; skew 500 ppm
Aug 10 18:08:04 dist-test-slave-27sr krb5kdc[1921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:08:04.181773 2540 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:08:04.182904 2540 webserver.cc:460] Webserver started at http://127.21.182.65:45777/ using document root <none> and password file <none>
I20260810 18:08:04.183518 2540 fs_manager.cc:359] Metadata directory not provided
I20260810 18:08:04.183615 2540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:08:04.183961 2540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:08:04 dist-test-slave-27sr krb5kdc[1921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:04.200641 2540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "bc1e58114ca04d4289f74e2dce0ab567"
format_stamp: "Formatted at 2026-08-10 18:08:04 on dist-test-slave-27sr"
server_key: "42f66fbbc616e0fb8abf0fc1ddcfda8a"
server_key_iv: "eb1b2c16288b02cf8aa7a642c71a00de"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:04.201589 2540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "bc1e58114ca04d4289f74e2dce0ab567"
format_stamp: "Formatted at 2026-08-10 18:08:04 on dist-test-slave-27sr"
server_key: "42f66fbbc616e0fb8abf0fc1ddcfda8a"
server_key_iv: "eb1b2c16288b02cf8aa7a642c71a00de"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:04.207515 2540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 18:08:04.230041 2558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:04.231997 2540 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260810 18:08:04.232192 2540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "bc1e58114ca04d4289f74e2dce0ab567"
format_stamp: "Formatted at 2026-08-10 18:08:04 on dist-test-slave-27sr"
server_key: "42f66fbbc616e0fb8abf0fc1ddcfda8a"
server_key_iv: "eb1b2c16288b02cf8aa7a642c71a00de"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:04.232403 2540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:08:04.256124 2540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:08:04.260478 2540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:08:04.260775 2540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:08:04.261643 2540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:08:04.262836 2540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:08:04.262940 2540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:04.263031 2540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:08:04.263083 2540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:04.274901 2540 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:37521
I20260810 18:08:04.276233 2540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:08:04.281247 2685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:37521 every 8 connection(s)
I20260810 18:08:04.285530 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 2540
I20260810 18:08:04.285691 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:08:04.286028 22233 external_mini_cluster.cc:1428] Setting key 68dc4591ec3ccad1a09525ebf7e5f0a0
Aug 10 18:08:04 dist-test-slave-27sr krb5kdc[1921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:08:04.294032 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:04.278945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:51733 (local address 127.21.182.126:33355)
0810 18:08:04.280583 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:04.280589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:04.281499 (+ 910us) server_negotiation.cc:407] Connection header received
0810 18:08:04.282168 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:04.282172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:04.282237 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:04.282372 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:08:04.283545 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:04.284236 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:04.285588 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:04.285947 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:04.286059 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:04.289244 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:08:04.289252 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:08:04.289257 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:08:04.289290 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:08:04.291528 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:04.292298 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:04.292302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:04.292304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:04.292405 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:04.292820 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:04.292827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:04.292829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:04.293019 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:08:04.293136 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:04.293692 (+ 556us) server_negotiation.cc:299] Negotiation successful
0810 18:08:04.293838 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1523,"thread_start_us":95,"threads_started":1}
I20260810 18:08:04.295066 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:04.279117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33355 (local address 127.21.182.65:51733)
0810 18:08:04.280895 (+ 1778us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:04.280956 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:04.281945 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:04.282573 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:04.282581 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:04.282883 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:08:04.283392 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:04.283406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:04.284410 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:04.284414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:04.285417 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:04.285430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:04.286260 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:04.286265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:04.286476 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:04.287219 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:08:04.287240 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:08:04.289032 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:08:04.291679 (+ 2647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:04.291689 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:04.291711 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:04.292165 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:04.292526 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:04.292529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:04.292531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:04.292686 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:04.293135 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:08:04.293144 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:08:04.293425 (+ 281us) client_negotiation.cc:769] Sending connection context
0810 18:08:04.293632 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 18:08:04.293980 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1514,"thread_start_us":183,"threads_started":1}
I20260810 18:08:04.296449 2686 heartbeater.cc:347] Connected to a master server at 127.21.182.126:33355
I20260810 18:08:04.296718 2686 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:04.297344 2686 heartbeater.cc:511] Master 127.21.182.126:33355 requested a full tablet report, sending...
I20260810 18:08:04.299315 2450 ts_manager.cc:194] Registered new tserver with Master: bc1e58114ca04d4289f74e2dce0ab567 (127.21.182.65:37521)
I20260810 18:08:04.300958 2450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:51733
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:08:04.328382 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.182.84:50209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33355
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:08:04 dist-test-slave-27sr krb5kdc[1921](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786385284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 18:08:04.510773 2692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:08:04.511108 2692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:08:04.511243 2692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:08:04.517156 2692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:08:04.517253 2692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:08:04.517369 2692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:08:04.525416 2692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.182.84:50209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:08:04.527048 2692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:08:04.528577 2692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:08:04.540345 2700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:04.540472 2702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:04.540345 2699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:04.541105 2692 server_base.cc:1034] running on GCE node
I20260810 18:08:04.541579 2692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:08:04.542346 2692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:08:04.543529 2692 hybrid_clock.cc:648] HybridClock initialized: now 1786385284543496 us; error 48 us; skew 500 ppm
Aug 10 18:08:04 dist-test-slave-27sr krb5kdc[1921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:08:04.546602 2692 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:08:04.547636 2692 webserver.cc:460] Webserver started at http://127.21.182.66:44471/ using document root <none> and password file <none>
I20260810 18:08:04.548395 2692 fs_manager.cc:359] Metadata directory not provided
I20260810 18:08:04.548488 2692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:08:04.548743 2692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:08:04 dist-test-slave-27sr krb5kdc[1921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:04.564246 2692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "4b23187460284cbea7f788cf6cf69006"
format_stamp: "Formatted at 2026-08-10 18:08:04 on dist-test-slave-27sr"
server_key: "c585ec7d1ee4d0ceeb8fa4bd5c23ee41"
server_key_iv: "5e1cd1d153e1c3db36dfc84987adb2f0"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:04.565742 2692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "4b23187460284cbea7f788cf6cf69006"
format_stamp: "Formatted at 2026-08-10 18:08:04 on dist-test-slave-27sr"
server_key: "c585ec7d1ee4d0ceeb8fa4bd5c23ee41"
server_key_iv: "5e1cd1d153e1c3db36dfc84987adb2f0"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:04.573647 2692 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.004s
I20260810 18:08:04.595233 2709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:04.597342 2692 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260810 18:08:04.597489 2692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "4b23187460284cbea7f788cf6cf69006"
format_stamp: "Formatted at 2026-08-10 18:08:04 on dist-test-slave-27sr"
server_key: "c585ec7d1ee4d0ceeb8fa4bd5c23ee41"
server_key_iv: "5e1cd1d153e1c3db36dfc84987adb2f0"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:04.597626 2692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:08:04.618899 2692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:08:04.626145 2692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:08:04.626449 2692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:08:04.627379 2692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:08:04.628814 2692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:08:04.628942 2692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:04.629050 2692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:08:04.629118 2692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:04.641960 2692 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:37811
I20260810 18:08:04.642869 2837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:37811 every 8 connection(s)
I20260810 18:08:04.643203 2692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:08:04.646438 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 2692
I20260810 18:08:04.646574 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:08:04.646888 22233 external_mini_cluster.cc:1428] Setting key efafc65734cefae4c1a58e977609c46b
Aug 10 18:08:04 dist-test-slave-27sr krb5kdc[1921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
I20260810 18:08:04.671697 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:04.645654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:40661 (local address 127.21.182.126:33355)
0810 18:08:04.645808 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:04.645814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:04.647835 (+ 2021us) server_negotiation.cc:407] Connection header received
0810 18:08:04.652965 (+ 5130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:04.652970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:04.653033 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:04.653088 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:08:04.655023 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:04.656069 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:04.658128 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:04.658522 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:04.658618 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:04.664789 (+ 6171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:08:04.664794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:08:04.664806 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:08:04.664859 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:08:04.667321 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:04.668340 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:04.668346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:04.668349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:04.668459 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:04.669058 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:04.669062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:04.669064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:04.669446 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:08:04.669673 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:04.671077 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0810 18:08:04.671305 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:08:04.674149 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:04.646194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33355 (local address 127.21.182.66:40661)
0810 18:08:04.647661 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:04.647724 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:04.652649 (+ 4925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:04.653676 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:04.653687 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:04.654072 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:08:04.654825 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:04.654846 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:04.656322 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:04.656326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:04.657961 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:04.657974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:04.658899 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:04.658903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:04.659137 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:04.661859 (+ 2722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:08:04.661905 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:08:04.664420 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:08:04.667603 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:04.667614 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:04.667638 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:04.668187 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:04.668603 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:04.668608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:04.668611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:04.668850 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:04.669787 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:08:04.669797 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:08:04.670588 (+ 791us) client_negotiation.cc:769] Sending connection context
0810 18:08:04.672236 (+ 1648us) client_negotiation.cc:240] Negotiation successful
0810 18:08:04.672860 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"thread_start_us":138,"threads_started":1}
I20260810 18:08:04.675662 2838 heartbeater.cc:347] Connected to a master server at 127.21.182.126:33355
I20260810 18:08:04.675968 2838 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:04.676590 2838 heartbeater.cc:511] Master 127.21.182.126:33355 requested a full tablet report, sending...
I20260810 18:08:04.677850 2450 ts_manager.cc:194] Registered new tserver with Master: 4b23187460284cbea7f788cf6cf69006 (127.21.182.66:37811)
I20260810 18:08:04.678619 2450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:40661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:08:04.706351 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.182.84:50209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33355
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:08:04 dist-test-slave-27sr krb5kdc[1921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
W20260810 18:08:04.900449 2845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:08:04.900985 2845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:08:04.901081 2845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:08:04.907274 2845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:08:04.907461 2845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:08:04.907648 2845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:08:04.916141 2845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.182.84:50209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:08:04.918082 2845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:08:04.919726 2845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:08:04.934659 2852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:04.934665 2854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:04.935490 2845 server_base.cc:1034] running on GCE node
W20260810 18:08:04.944567 2851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:04.945484 2845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:08:04.946528 2845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:08:04.954062 2845 hybrid_clock.cc:648] HybridClock initialized: now 1786385284954002 us; error 69 us; skew 500 ppm
Aug 10 18:08:04 dist-test-slave-27sr krb5kdc[1921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:08:04.958693 2845 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:08:04.960155 2845 webserver.cc:460] Webserver started at http://127.21.182.67:38661/ using document root <none> and password file <none>
I20260810 18:08:04.961030 2845 fs_manager.cc:359] Metadata directory not provided
I20260810 18:08:04.961205 2845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:08:04.961546 2845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:08:04 dist-test-slave-27sr krb5kdc[1921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:04.989742 2845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "f94144696c514451afd1f0d4980703f5"
format_stamp: "Formatted at 2026-08-10 18:08:04 on dist-test-slave-27sr"
server_key: "e9482313bb71ed020fbe32f048cb1ad3"
server_key_iv: "55da5f02d0f43a39bfe941a8ebe9aaaa"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:04.990644 2845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "f94144696c514451afd1f0d4980703f5"
format_stamp: "Formatted at 2026-08-10 18:08:04 on dist-test-slave-27sr"
server_key: "e9482313bb71ed020fbe32f048cb1ad3"
server_key_iv: "55da5f02d0f43a39bfe941a8ebe9aaaa"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:04.996475 2845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 18:08:05.015492 2863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:05.017213 2845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:08:05.017359 2845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "f94144696c514451afd1f0d4980703f5"
format_stamp: "Formatted at 2026-08-10 18:08:04 on dist-test-slave-27sr"
server_key: "e9482313bb71ed020fbe32f048cb1ad3"
server_key_iv: "55da5f02d0f43a39bfe941a8ebe9aaaa"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:05.017493 2845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:08:05.037863 2845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:08:05.041888 2845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:08:05.042140 2845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:08:05.042900 2845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:08:05.044091 2845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:08:05.044207 2845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:05.044322 2845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:08:05.044394 2845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:05.061890 2845 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:34515
I20260810 18:08:05.063243 2845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:08:05.064043 2991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:34515 every 8 connection(s)
I20260810 18:08:05.068136 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 2845
I20260810 18:08:05.068284 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:08:05.068596 22233 external_mini_cluster.cc:1428] Setting key c3620939915bc728259418da62e130f9
Aug 10 18:08:05 dist-test-slave-27sr krb5kdc[1921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:08:05.083163 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.066529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:49689 (local address 127.21.182.126:33355)
0810 18:08:05.066665 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:05.066669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:05.067920 (+ 1251us) server_negotiation.cc:407] Connection header received
0810 18:08:05.069450 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:05.069454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:05.069521 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:05.069585 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:08:05.071563 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.072620 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.074551 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.074914 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.074996 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:05.078483 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:08:05.078487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:08:05.078490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:08:05.078517 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:08:05.080492 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:05.081223 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:05.081227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:05.081229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:05.081327 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:05.081811 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:05.081814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:05.081816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:05.082045 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:08:05.082151 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:05.082863 (+ 712us) server_negotiation.cc:299] Negotiation successful
0810 18:08:05.082997 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:08:05.084398 2994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.066847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33355 (local address 127.21.182.67:49689)
0810 18:08:05.067624 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:05.067684 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:05.068978 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:05.070082 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:05.070092 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:05.070498 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:08:05.071367 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:05.071390 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.072817 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.072821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:05.074390 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:05.074402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.075228 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.075232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:05.075444 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:05.076322 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:08:05.076347 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:08:05.078257 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:08:05.080636 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:05.080644 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:05.080697 (+ 53us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:05.081091 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:05.081446 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:05.081450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:05.081452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:05.081686 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:05.082180 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:08:05.082188 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:08:05.082543 (+ 355us) client_negotiation.cc:769] Sending connection context
0810 18:08:05.082798 (+ 255us) client_negotiation.cc:240] Negotiation successful
0810 18:08:05.083183 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":159,"threads_started":1}
I20260810 18:08:05.085826 2992 heartbeater.cc:347] Connected to a master server at 127.21.182.126:33355
I20260810 18:08:05.086105 2992 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:05.088063 2992 heartbeater.cc:511] Master 127.21.182.126:33355 requested a full tablet report, sending...
I20260810 18:08:05.089372 2450 ts_manager.cc:194] Registered new tserver with Master: f94144696c514451afd1f0d4980703f5 (127.21.182.67:34515)
I20260810 18:08:05.090127 2450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:49689
I20260810 18:08:05.100263 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:08:05.113070 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.102873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.21.182.126:33355)
0810 18:08:05.103038 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:05.103042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:05.103269 (+ 227us) server_negotiation.cc:407] Connection header received
0810 18:08:05.103438 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:05.103441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:05.103496 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:05.103577 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:08:05.104726 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.105543 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.106906 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.107197 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.107280 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:05.109197 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:08:05.109201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:08:05.109204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:08:05.109233 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:08:05.111091 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:05.111627 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:05.111631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:05.111633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:05.111729 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:05.112077 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:05.112080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:05.112082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:05.112307 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:08:05.112438 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:05.112776 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 18:08:05.112898 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 18:08:05.116500 2450 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:08:05.120826 2450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:08:05.140892 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.131992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:37811 (local address 127.0.0.1:47806)
0810 18:08:05.133211 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:05.133262 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:05.133390 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:05.134175 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:05.134182 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:05.134206 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:08:05.134357 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:05.134363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.136682 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.136688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:05.139149 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:05.139160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.140332 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.140336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:05.140458 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:05.140502 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 18:08:05.140593 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 18:08:05.140727 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"thread_start_us":109,"threads_started":1}
I20260810 18:08:05.144090 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.132722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47806 (local address 127.21.182.66:37811)
0810 18:08:05.133656 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:05.133685 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:05.133709 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:08:05.133785 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:05.133790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:05.134000 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:05.134507 (+ 507us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:08:05.134633 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.136529 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.139339 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.140192 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.143549 (+ 3357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:05.143745 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:05.143807 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 18:08:05.143943 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":84,"threads_started":1}
I20260810 18:08:05.144819 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.133778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:37521 (local address 127.0.0.1:54940)
0810 18:08:05.137160 (+ 3382us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:05.137173 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:05.137300 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:05.138212 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:05.138216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:05.138236 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:08:05.138409 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:05.138417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.141617 (+ 3200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.141620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:05.142831 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:05.142843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.144357 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.144361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:05.144463 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:05.144489 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:08:05.144572 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 18:08:05.144669 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3319,"thread_start_us":88,"threads_started":1}
I20260810 18:08:05.145570 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.134779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.21.182.65:37521)
0810 18:08:05.135779 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:05.135785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:05.137398 (+ 1613us) server_negotiation.cc:407] Connection header received
0810 18:08:05.137482 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:05.137487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:05.137832 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:05.138004 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:08:05.138560 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.141470 (+ 2910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.142985 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.143848 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.144136 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:05.144291 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:05.145313 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0810 18:08:05.145425 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":88,"threads_started":1}
I20260810 18:08:05.146595 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.132681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34515 (local address 127.0.0.1:45862)
0810 18:08:05.135390 (+ 2709us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:05.135407 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:05.135528 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:05.136879 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:05.136883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:05.136901 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:08:05.136993 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:05.136999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.140708 (+ 3709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.140712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:05.143366 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:05.143378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.146220 (+ 2842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.146224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:05.146324 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:05.146347 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:08:05.146410 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 18:08:05.146473 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2635,"thread_start_us":134,"threads_started":1}
I20260810 18:08:05.147432 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.133189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.21.182.67:34515)
0810 18:08:05.136155 (+ 2966us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:05.136187 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:05.136211 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:08:05.136285 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:05.136290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:05.136499 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:05.136712 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:08:05.139296 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.140542 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.144409 (+ 3867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.145259 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.146985 (+ 1726us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:05.147145 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:05.147199 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 18:08:05.147292 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2856,"thread_start_us":156,"threads_started":1}
I20260810 18:08:05.149225 2613 tablet_service.cc:1467] Processing CreateTablet for tablet b3a3310ed67c48999f9eef79101914f7 (DEFAULT_TABLE table=test-table [id=a080240250b34dd2b67ce2cbd6e5592a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:08:05.149533 2755 tablet_service.cc:1467] Processing CreateTablet for tablet b3a3310ed67c48999f9eef79101914f7 (DEFAULT_TABLE table=test-table [id=a080240250b34dd2b67ce2cbd6e5592a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:08:05.150482 2613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a3310ed67c48999f9eef79101914f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:08:05.150604 2755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a3310ed67c48999f9eef79101914f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:08:05.154349 2918 tablet_service.cc:1467] Processing CreateTablet for tablet b3a3310ed67c48999f9eef79101914f7 (DEFAULT_TABLE table=test-table [id=a080240250b34dd2b67ce2cbd6e5592a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:08:05.155784 2918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a3310ed67c48999f9eef79101914f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:08:05.164388 3011 tablet_bootstrap.cc:492] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006: Bootstrap starting.
I20260810 18:08:05.167263 3011 tablet_bootstrap.cc:654] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006: Neither blocks nor log segments found. Creating new log.
I20260810 18:08:05.167337 3010 tablet_bootstrap.cc:492] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567: Bootstrap starting.
I20260810 18:08:05.168861 3011 log.cc:824] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006: Log is configured to *not* fsync() on all Append() calls
I20260810 18:08:05.170440 3010 tablet_bootstrap.cc:654] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567: Neither blocks nor log segments found. Creating new log.
I20260810 18:08:05.170989 3013 tablet_bootstrap.cc:492] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5: Bootstrap starting.
I20260810 18:08:05.172699 3011 tablet_bootstrap.cc:492] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006: No bootstrap required, opened a new log
I20260810 18:08:05.172716 3010 log.cc:824] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567: Log is configured to *not* fsync() on all Append() calls
I20260810 18:08:05.172960 3011 ts_tablet_manager.cc:1397] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 18:08:05.176172 3013 tablet_bootstrap.cc:654] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5: Neither blocks nor log segments found. Creating new log.
I20260810 18:08:05.177186 3013 log.cc:824] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:08:05.177285 3011 raft_consensus.cc:348] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b23187460284cbea7f788cf6cf69006" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37811 } } peers { permanent_uuid: "f94144696c514451afd1f0d4980703f5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34515 } } peers { permanent_uuid: "bc1e58114ca04d4289f74e2dce0ab567" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37521 } }
I20260810 18:08:05.177987 3011 raft_consensus.cc:374] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:08:05.179371 3011 raft_consensus.cc:729] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b23187460284cbea7f788cf6cf69006, State: Initialized, Role: FOLLOWER
I20260810 18:08:05.179966 3010 tablet_bootstrap.cc:492] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567: No bootstrap required, opened a new log
I20260810 18:08:05.180039 3011 consensus_queue.cc:260] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b23187460284cbea7f788cf6cf69006" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37811 } } peers { permanent_uuid: "f94144696c514451afd1f0d4980703f5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34515 } } peers { permanent_uuid: "bc1e58114ca04d4289f74e2dce0ab567" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37521 } }
I20260810 18:08:05.180364 3010 ts_tablet_manager.cc:1397] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.002s
I20260810 18:08:05.181182 3011 ts_tablet_manager.cc:1428] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260810 18:08:05.181625 2838 heartbeater.cc:503] Master 127.21.182.126:33355 was elected leader, sending a full tablet report...
I20260810 18:08:05.186497 3010 raft_consensus.cc:348] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b23187460284cbea7f788cf6cf69006" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37811 } } peers { permanent_uuid: "f94144696c514451afd1f0d4980703f5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34515 } } peers { permanent_uuid: "bc1e58114ca04d4289f74e2dce0ab567" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37521 } }
I20260810 18:08:05.187186 3013 tablet_bootstrap.cc:492] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5: No bootstrap required, opened a new log
I20260810 18:08:05.187458 3013 ts_tablet_manager.cc:1397] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20260810 18:08:05.191045 3010 raft_consensus.cc:374] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:08:05.191331 3010 raft_consensus.cc:729] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc1e58114ca04d4289f74e2dce0ab567, State: Initialized, Role: FOLLOWER
I20260810 18:08:05.192409 3010 consensus_queue.cc:260] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b23187460284cbea7f788cf6cf69006" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37811 } } peers { permanent_uuid: "f94144696c514451afd1f0d4980703f5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34515 } } peers { permanent_uuid: "bc1e58114ca04d4289f74e2dce0ab567" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37521 } }
I20260810 18:08:05.192682 3019 raft_consensus.cc:482] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:08:05.192859 3019 raft_consensus.cc:504] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b23187460284cbea7f788cf6cf69006" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37811 } } peers { permanent_uuid: "f94144696c514451afd1f0d4980703f5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34515 } } peers { permanent_uuid: "bc1e58114ca04d4289f74e2dce0ab567" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37521 } }
I20260810 18:08:05.193783 3013 raft_consensus.cc:348] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b23187460284cbea7f788cf6cf69006" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37811 } } peers { permanent_uuid: "f94144696c514451afd1f0d4980703f5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34515 } } peers { permanent_uuid: "bc1e58114ca04d4289f74e2dce0ab567" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37521 } }
I20260810 18:08:05.195488 3019 leader_election.cc:288] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b23187460284cbea7f788cf6cf69006 (127.21.182.66:37811), f94144696c514451afd1f0d4980703f5 (127.21.182.67:34515)
I20260810 18:08:05.195936 2686 heartbeater.cc:503] Master 127.21.182.126:33355 was elected leader, sending a full tablet report...
I20260810 18:08:05.196192 3010 ts_tablet_manager.cc:1428] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567: Time spent starting tablet: real 0.016s user 0.004s sys 0.003s
I20260810 18:08:05.195968 3013 raft_consensus.cc:374] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:08:05.196370 3013 raft_consensus.cc:729] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94144696c514451afd1f0d4980703f5, State: Initialized, Role: FOLLOWER
I20260810 18:08:05.200205 3013 consensus_queue.cc:260] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b23187460284cbea7f788cf6cf69006" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37811 } } peers { permanent_uuid: "f94144696c514451afd1f0d4980703f5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34515 } } peers { permanent_uuid: "bc1e58114ca04d4289f74e2dce0ab567" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37521 } }
I20260810 18:08:05.201167 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.194497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:43903 (local address 127.21.182.66:37811)
0810 18:08:05.194620 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:05.194624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:05.196034 (+ 1410us) server_negotiation.cc:407] Connection header received
0810 18:08:05.196258 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:05.196261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:05.196312 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:05.196361 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:08:05.196826 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.197616 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.199442 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.200179 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.200276 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:05.200331 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:05.200952 (+ 621us) server_negotiation.cc:299] Negotiation successful
0810 18:08:05.201037 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:08:05.202571 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.194433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:37811 (local address 127.21.182.65:43903)
0810 18:08:05.195941 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:05.195957 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:05.196127 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:05.196518 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:05.196521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:05.196539 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:08:05.196695 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:05.196701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.197906 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.197910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:05.199289 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:05.199301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.200443 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.200447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:05.200548 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:05.200572 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:08:05.201498 (+ 926us) client_negotiation.cc:240] Negotiation successful
0810 18:08:05.202421 (+ 923us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1416,"thread_start_us":122,"threads_started":1}
I20260810 18:08:05.201969 3013 ts_tablet_manager.cc:1428] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260810 18:08:05.203194 2772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a3310ed67c48999f9eef79101914f7" candidate_uuid: "bc1e58114ca04d4289f74e2dce0ab567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b23187460284cbea7f788cf6cf69006" is_pre_election: true
I20260810 18:08:05.203255 2992 heartbeater.cc:503] Master 127.21.182.126:33355 was elected leader, sending a full tablet report...
I20260810 18:08:05.203482 2772 raft_consensus.cc:2454] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc1e58114ca04d4289f74e2dce0ab567 in term 0.
I20260810 18:08:05.204164 2559 leader_election.cc:302] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b23187460284cbea7f788cf6cf69006, bc1e58114ca04d4289f74e2dce0ab567; no voters:
I20260810 18:08:05.204456 3019 raft_consensus.cc:2781] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:08:05.204530 3019 raft_consensus.cc:482] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:08:05.204571 3019 raft_consensus.cc:3037] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:08:05.206650 3019 raft_consensus.cc:504] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b23187460284cbea7f788cf6cf69006" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37811 } } peers { permanent_uuid: "f94144696c514451afd1f0d4980703f5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34515 } } peers { permanent_uuid: "bc1e58114ca04d4289f74e2dce0ab567" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37521 } }
I20260810 18:08:05.207336 2772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a3310ed67c48999f9eef79101914f7" candidate_uuid: "bc1e58114ca04d4289f74e2dce0ab567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b23187460284cbea7f788cf6cf69006"
I20260810 18:08:05.207479 2772 raft_consensus.cc:3037] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:08:05.208806 2772 raft_consensus.cc:2454] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc1e58114ca04d4289f74e2dce0ab567 in term 1.
I20260810 18:08:05.209167 3019 leader_election.cc:288] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [CANDIDATE]: Term 1 election: Requested vote from peers 4b23187460284cbea7f788cf6cf69006 (127.21.182.66:37811), f94144696c514451afd1f0d4980703f5 (127.21.182.67:34515)
I20260810 18:08:05.209185 2559 leader_election.cc:302] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b23187460284cbea7f788cf6cf69006, bc1e58114ca04d4289f74e2dce0ab567; no voters:
I20260810 18:08:05.209389 3019 raft_consensus.cc:2781] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:08:05.209491 3019 raft_consensus.cc:686] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [term 1 LEADER]: Becoming Leader. State: Replica: bc1e58114ca04d4289f74e2dce0ab567, State: Running, Role: LEADER
I20260810 18:08:05.209837 3019 consensus_queue.cc:237] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b23187460284cbea7f788cf6cf69006" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37811 } } peers { permanent_uuid: "f94144696c514451afd1f0d4980703f5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34515 } } peers { permanent_uuid: "bc1e58114ca04d4289f74e2dce0ab567" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37521 } }
I20260810 18:08:05.213311 2448 catalog_manager.cc:5495] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc1e58114ca04d4289f74e2dce0ab567 (127.21.182.65). New cstate: current_term: 1 leader_uuid: "bc1e58114ca04d4289f74e2dce0ab567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b23187460284cbea7f788cf6cf69006" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f94144696c514451afd1f0d4980703f5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc1e58114ca04d4289f74e2dce0ab567" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 37521 } health_report { overall_health: HEALTHY } } }
I20260810 18:08:05.218595 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.195342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:60521 (local address 127.21.182.67:34515)
0810 18:08:05.195454 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:05.195459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:05.200868 (+ 5409us) server_negotiation.cc:407] Connection header received
0810 18:08:05.202172 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:05.202176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:05.202248 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:05.202301 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:08:05.214804 (+ 12503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.215598 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.217256 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.217870 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.218000 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:05.218056 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:05.218379 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 18:08:05.218466 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:08:05.219146 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.194990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34515 (local address 127.21.182.65:60521)
0810 18:08:05.200782 (+ 5792us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:05.200796 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:05.201401 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:05.214421 (+ 13020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:05.214425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:05.214448 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:08:05.214575 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:05.214581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.215792 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.215796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:05.217103 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:05.217115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.218128 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.218132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:05.218226 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:05.218258 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 18:08:05.218914 (+ 656us) client_negotiation.cc:240] Negotiation successful
0810 18:08:05.219001 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5724,"thread_start_us":106,"threads_started":1}
I20260810 18:08:05.221685 2938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a3310ed67c48999f9eef79101914f7" candidate_uuid: "bc1e58114ca04d4289f74e2dce0ab567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f94144696c514451afd1f0d4980703f5"
I20260810 18:08:05.221706 2924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a3310ed67c48999f9eef79101914f7" candidate_uuid: "bc1e58114ca04d4289f74e2dce0ab567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f94144696c514451afd1f0d4980703f5" is_pre_election: true
I20260810 18:08:05.221988 2938 raft_consensus.cc:3037] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:08:05.223345 2938 raft_consensus.cc:2454] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc1e58114ca04d4289f74e2dce0ab567 in term 1.
Aug 10 18:08:05 dist-test-slave-27sr krb5kdc[1921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:05.233373 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.228106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.21.182.65:37521)
0810 18:08:05.228231 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:05.228235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:05.228248 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:08:05.228327 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:05.228330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:05.228395 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:05.228500 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:08:05.228967 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.229730 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.230976 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:05.231317 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:05.231423 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:05.231955 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:08:05.232559 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:08:05.232691 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:05.233164 (+ 473us) server_negotiation.cc:299] Negotiation successful
0810 18:08:05.233229 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:08:05.241304 2772 raft_consensus.cc:1261] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006 [term 1 FOLLOWER]: Refusing update from remote peer bc1e58114ca04d4289f74e2dce0ab567: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:08:05.242134 3021 consensus_queue.cc:1035] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b23187460284cbea7f788cf6cf69006" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:08:05.242856 2938 raft_consensus.cc:1261] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5 [term 1 FOLLOWER]: Refusing update from remote peer bc1e58114ca04d4289f74e2dce0ab567: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:08:05.243441 3021 consensus_queue.cc:1035] T b3a3310ed67c48999f9eef79101914f7 P bc1e58114ca04d4289f74e2dce0ab567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f94144696c514451afd1f0d4980703f5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:08:05.255035 3028 mvcc.cc:205] Tried to move back new op lower bound from 7317034128337850368 to 7317034128223117312. Current Snapshot: MvccSnapshot[applied={T|T < 7317034128337850368}]
I20260810 18:08:05.255960 3031 mvcc.cc:205] Tried to move back new op lower bound from 7317034128337850368 to 7317034128223117312. Current Snapshot: MvccSnapshot[applied={T|T < 7317034128337850368}]
I20260810 18:08:05.259407 3029 mvcc.cc:205] Tried to move back new op lower bound from 7317034128337850368 to 7317034128223117312. Current Snapshot: MvccSnapshot[applied={T|T < 7317034128337850368}]
I20260810 18:08:05.271353 2448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:08:05.271569 2448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:08:05.279995 2448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08f19d2592e04c2fb8e6cad2352671f0: Sending DeleteTablet for 3 replicas of tablet b3a3310ed67c48999f9eef79101914f7
I20260810 18:08:05.281002 2755 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a3310ed67c48999f9eef79101914f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:08:05 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:47806
I20260810 18:08:05.282233 2918 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a3310ed67c48999f9eef79101914f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:08:05 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:45862
I20260810 18:08:05.283259 2613 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a3310ed67c48999f9eef79101914f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:08:05 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:54940
I20260810 18:08:05.284735 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 2540
I20260810 18:08:05.287365 3039 tablet_replica.cc:330] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006: stopping tablet replica
I20260810 18:08:05.287755 3039 raft_consensus.cc:2229] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:08:05.288082 3039 raft_consensus.cc:2258] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:08:05.288486 3040 tablet_replica.cc:330] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5: stopping tablet replica
I20260810 18:08:05.288784 3040 raft_consensus.cc:2229] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:08:05.289076 3040 raft_consensus.cc:2258] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:08:05.291240 3040 ts_tablet_manager.cc:1905] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:08:05.294271 3040 ts_tablet_manager.cc:1918] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:08:05.294380 3040 log.cc:1196] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/b3a3310ed67c48999f9eef79101914f7
I20260810 18:08:05.294832 3040 ts_tablet_manager.cc:1939] T b3a3310ed67c48999f9eef79101914f7 P f94144696c514451afd1f0d4980703f5: Deleting consensus metadata
I20260810 18:08:05.297163 2432 catalog_manager.cc:4841] TS f94144696c514451afd1f0d4980703f5 (127.21.182.67:34515): tablet b3a3310ed67c48999f9eef79101914f7 (table test-table [id=a080240250b34dd2b67ce2cbd6e5592a]) successfully deleted
I20260810 18:08:05.297873 3039 ts_tablet_manager.cc:1905] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:08:05.301642 3039 ts_tablet_manager.cc:1918] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:08:05.301775 3039 log.cc:1196] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/b3a3310ed67c48999f9eef79101914f7
I20260810 18:08:05.302117 3039 ts_tablet_manager.cc:1939] T b3a3310ed67c48999f9eef79101914f7 P 4b23187460284cbea7f788cf6cf69006: Deleting consensus metadata
W20260810 18:08:05.304740 2431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:37521 (error 108)
I20260810 18:08:05.305588 2430 catalog_manager.cc:4841] TS 4b23187460284cbea7f788cf6cf69006 (127.21.182.66:37811): tablet b3a3310ed67c48999f9eef79101914f7 (table test-table [id=a080240250b34dd2b67ce2cbd6e5592a]) successfully deleted
I20260810 18:08:05.306888 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 2692
I20260810 18:08:05.306923 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:05.306595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:37521 (local address 127.0.0.1:54950)
0810 18:08:05.306756 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:05.306832 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:37521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 18:08:05.333165 2431 catalog_manager.cc:4584] TS bc1e58114ca04d4289f74e2dce0ab567 (127.21.182.65:37521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3a3310ed67c48999f9eef79101914f7: Network error: Client connection negotiation failed: client connection to 127.21.182.65:37521: connect: Connection refused (error 111)
I20260810 18:08:05.346743 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 2845
I20260810 18:08:05.364192 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 2412
I20260810 18:08:05.378054 22233 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 18:08:05.724252 22233 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 18:08:05.724426 22233 mini_ranger.cc:66] Stopping Ranger...
I20260810 18:08:06.070287 22233 mini_ranger.cc:68] Stopped Ranger
2026-08-10 18:08:06.070 UTC [1952] LOG: received smart shutdown request
2026-08-10 18:08:06.074 UTC [1952] LOG: background worker "logical replication launcher" (PID 1961) exited with exit code 1
2026-08-10 18:08:06.075 UTC [1956] LOG: shutting down
2026-08-10 18:08:06.267 UTC [1952] LOG: database system is shut down
2026-08-10T18:08:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50461 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:08:06 dist-test-slave-27sr krb5kdc[3055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:08:06 dist-test-slave-27sr krb5kdc[3055](info): set up 2 sockets
Aug 10 18:08:06 dist-test-slave-27sr krb5kdc[3055](info): commencing operation
krb5kdc: starting...
W20260810 18:08:07.568984 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:08:07 dist-test-slave-27sr krb5kdc[3055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:08:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:08:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/rangeradmin_127.21.182.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/rangeradmin_127.21.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/rangerlookup_127.21.182.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/rangerlookup_127.21.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/HTTP_127.21.182.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/HTTP_127.21.182.84@KRBTEST.COM.keytab.
I20260810 18:08:07.755420 22233 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0GlVnd/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/postgres -l logfile start
2026-08-10 18:08:10.485 UTC [3089] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 18:08:10.485 UTC [3089] LOG: listening on IPv4 address "127.21.182.84", port 42527
2026-08-10 18:08:10.488 UTC [3089] LOG: listening on Unix socket "/tmp/.s.PGSQL.42527"
2026-08-10 18:08:10.510 UTC [3093] LOG: database system was shut down at 2026-08-10 18:08:08 UTC
2026-08-10 18:08:10.514 UTC [3089] LOG: database system is ready to accept connections
I20260810 18:08:11.581260 22233 mini_postgres.cc:96] Postgres bound to 42527
2026-08-10 18:08:11.590 UTC [3103] FATAL: database "slave" does not exist
127.21.182.84:42527 - accepting connections
I20260810 18:08:11.590853 22233 mini_ranger.cc:161] Starting Ranger...
I20260810 18:08:11.610065 22233 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 18:08:12.167770 22233 mini_ranger.cc:87] Created ranger Postgres database
I20260810 18:08:12.167856 22233 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-admin
2026-08-10 18:08:12,485 [I] DB FLAVOR :POSTGRES
2026-08-10 18:08:12,485 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 18:08:12,486 [I] Checking connection..
2026-08-10 18:08:12,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 18:08:12,740 [I] Checking connection passed.
2026-08-10 18:08:12,740 [I] --------- Verifying version history table ---------
2026-08-10 18:08:12,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:08:13,002 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 18:08:13,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:08:13,285 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 18:08:13,285 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 18:08:13,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 18:08:13.548 UTC [3179] WARNING: there is no transaction in progress
2026-08-10 18:08:13,557 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 18:08:13,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:08:13,806 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 18:08:13,806 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 18:08:13,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 18:08:14,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 18:08:14,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-27sr.c.gcp-upstream.internal','N') ;"
2026-08-10 18:08:14,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 18:08:14,831 [I] Table x_portal_user does not exist in database ranger
2026-08-10 18:08:14,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 18:08:15,113 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 18:08:15,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 18:08:15,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 18:08:15,634 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 18:08:15,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 18:08:16.645 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.661 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.679 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.696 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.713 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.730 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.786 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.794 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.810 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.821 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.833 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.847 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.858 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.865 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.873 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:16.882 UTC [3300] WARNING: there is no transaction in progress
2026-08-10 18:08:17,581 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 18:08:17,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 18:08:17,849 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 18:08:17,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 18:08:18,108 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 18:08:18,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 18:08:18,376 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 18:08:18,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 18:08:18,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 18:08:18,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-27sr.c.gcp-upstream.internal';"
2026-08-10 18:08:19,102 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 18:08:19,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 18:08:19,363 [I] DB_PATCHES have already been applied
I20260810 18:08:19.369462 22233 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-admin:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0GlVnd/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 18:08:19.369535 22233 mini_ranger.cc:193] Using host: 127.21.182.84
I20260810 18:08:19.374150 22233 mini_ranger.cc:227] Ranger admin URL: http://127.21.182.84:53763
Aug 10, 2026 6:08:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:08:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:08:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 6:08:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 6:08:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 6:08:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 6:08:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 18:08:20.618412 22233 mini_ranger.cc:160] Time spent starting Ranger: real 9.028s user 0.005s sys 0.009s
Aug 10, 2026 6:08:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 6:08:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 6:08:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 6:08:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 6:08:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 6:08:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 6:08:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 6:08:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 6:08:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 6:08:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 6:08:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 6:08:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 6:08:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 6:08:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 6:08:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 6:08:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 6:08:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 6:08:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 18:08:45.543989 22233 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.182.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.182.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/rangerkms_127.21.182.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.182.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/rangerkms_127.21.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.182.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.182.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.182.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/HTTP_127.21.182.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.182.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/HTTP_127.21.182.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 18:08:45 dist-test-slave-27sr krb5kdc[3055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385325, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 18:08:45.645964 22233 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 18:08:45.657445 22233 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 18:08:46.720570 22233 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 18:08:46.720649 22233 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-kms
I20260810 18:08:46.720662 22233 mini_ranger_kms.cc:227] Using postgres at 127.21.182.84:42527
2026-08-10 18:08:46,933 [I] DB FLAVOR :POSTGRES
2026-08-10 18:08:46,934 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 18:08:46,934 [I] Checking connection
2026-08-10 18:08:46,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 18:08:47,202 [I] connection success
2026-08-10 18:08:47,203 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 18:08:47,203 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 18:08:47,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 18:08:47,456 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 18:08:47,456 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 18:08:47,456 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 18:08:47,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.182.84:42527/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 18:08:47,737 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 18:08:48.240285 22233 mini_ranger_kms.cc:310] Created kms service
I20260810 18:08:48.430047 22233 mini_ranger_kms.cc:326] Created kudu user
I20260810 18:08:48.483016 22233 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 18:08:48 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385325, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:48.942482 22233 mini_ranger_kms.cc:384] Added ranger policy
I20260810 18:08:48.942754 22233 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-kms:/tmp/dist-test-task0GlVnd/build/debug/bin/postgresql.jar:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0GlVnd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 18:08:48.942783 22233 mini_ranger_kms.cc:240] Using host: 127.21.182.84
I20260810 18:08:48.948473 22233 mini_ranger_kms.cc:276] Ranger KMS PID: 3495
I20260810 18:08:48.948576 22233 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.182.84:60865
Aug 10, 2026 6:08:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:08:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:08:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 6:08:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 6:08:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 6:08:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 18:08:51 dist-test-slave-27sr krb5kdc[3055](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786385331, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.182.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:08:51.532564 22233 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.887s user 0.025s sys 0.172s
I20260810 18:08:51.532692 22233 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 18:08:51.532757 22233 mini_ranger_kms.cc:401] 127.21.182.84:60865/kms/v1/keys
Aug 10 18:08:51 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385331, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.182.84@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
Aug 10 18:08:51 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385331, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.182.84@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
Aug 10 18:08:54 dist-test-slave-27sr krb5kdc[3055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:08:54.276461 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:43349
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:43349
--ranger_config_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.182.84:60865/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:08:54.404668 3553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:08:54.405007 3553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:08:54.405087 3553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:08:54.409324 3553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:08:54.409405 3553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:08:54.409425 3553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:08:54.409442 3553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:08:54.409497 3553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:08:54.415419 3553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.182.84:60865/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:43349
--ranger_config_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:43349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:08:54.416735 3553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:08:54.417935 3553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:08:54.427732 3553 server_base.cc:1034] running on GCE node
W20260810 18:08:54.427675 3559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:54.427671 3561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:54.427671 3558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:54.428609 3553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:08:54.429943 3553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:08:54.431146 3553 hybrid_clock.cc:648] HybridClock initialized: now 1786385334431130 us; error 29 us; skew 500 ppm
Aug 10 18:08:54 dist-test-slave-27sr krb5kdc[3055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385334, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:08:54.434312 3553 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:08:54.435413 3553 webserver.cc:460] Webserver started at http://127.21.182.126:33145/ using document root <none> and password file <none>
I20260810 18:08:54.436133 3553 fs_manager.cc:359] Metadata directory not provided
I20260810 18:08:54.436200 3553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:08:54.436443 3553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:08:54 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385334, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:54.497656 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "4c1215894a714060a248ef8bea1c1355"
format_stamp: "Formatted at 2026-08-10 18:08:54 on dist-test-slave-27sr"
server_key: "b355d673c6ad64e8426fa33277f3384e"
server_key_iv: "43835f8f040a371f848e1a934fb8525d"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:54.498517 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "4c1215894a714060a248ef8bea1c1355"
format_stamp: "Formatted at 2026-08-10 18:08:54 on dist-test-slave-27sr"
server_key: "b355d673c6ad64e8426fa33277f3384e"
server_key_iv: "43835f8f040a371f848e1a934fb8525d"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:54.504189 3553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260810 18:08:54.523005 3568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:54.524602 3553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:08:54.524742 3553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "4c1215894a714060a248ef8bea1c1355"
format_stamp: "Formatted at 2026-08-10 18:08:54 on dist-test-slave-27sr"
server_key: "b355d673c6ad64e8426fa33277f3384e"
server_key_iv: "43835f8f040a371f848e1a934fb8525d"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:54.524868 3553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:08:54.542445 3553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:08:54.545291 3553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:08:54.545468 3553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:08:54.553200 3553 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:43349
I20260810 18:08:54.553226 3625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:43349 every 8 connection(s)
I20260810 18:08:54.554266 3553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:08:54.556202 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 3553
I20260810 18:08:54.556299 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:08:54.556638 22233 external_mini_cluster.cc:1428] Setting key 997ffc59ec874ec2684589185dd91264
I20260810 18:08:54.557809 3626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:08:54.565205 3626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355: Bootstrap starting.
Aug 10 18:08:54 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:08:54.567869 3626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355: Neither blocks nor log segments found. Creating new log.
I20260810 18:08:54.568789 3626 log.cc:824] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355: Log is configured to *not* fsync() on all Append() calls
I20260810 18:08:54.571215 3626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355: No bootstrap required, opened a new log
I20260810 18:08:54.572930 3629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:54.558210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47472 (local address 127.21.182.126:43349)
0810 18:08:54.558655 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:54.558671 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:54.558709 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:08:54.559362 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:54.559379 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:54.559745 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:54.560111 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:08:54.561120 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:54.562127 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:54.563313 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:54.563786 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:54.563997 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:54.566938 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:08:54.566945 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:08:54.566960 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:08:54.566991 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:08:54.569426 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:54.570007 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:54.570012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:54.570018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:54.570125 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:54.570462 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:54.570468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:54.570473 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:54.571063 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:08:54.571260 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:54.571829 (+ 569us) server_negotiation.cc:299] Negotiation successful
0810 18:08:54.572159 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":150,"threads_started":1}
I20260810 18:08:54.574927 3626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1215894a714060a248ef8bea1c1355" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43349 } }
I20260810 18:08:54.575223 3626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:08:54.575320 3626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1215894a714060a248ef8bea1c1355, State: Initialized, Role: FOLLOWER
I20260810 18:08:54.575830 3626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1215894a714060a248ef8bea1c1355" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43349 } }
I20260810 18:08:54.576072 3626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:08:54.576141 3626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:08:54.576248 3626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:08:54.577235 3626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1215894a714060a248ef8bea1c1355" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43349 } }
I20260810 18:08:54.577610 3626 leader_election.cc:302] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c1215894a714060a248ef8bea1c1355; no voters:
I20260810 18:08:54.577955 3626 leader_election.cc:288] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:08:54.578150 3632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:08:54.578399 3632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 1 LEADER]: Becoming Leader. State: Replica: 4c1215894a714060a248ef8bea1c1355, State: Running, Role: LEADER
I20260810 18:08:54.578768 3632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1215894a714060a248ef8bea1c1355" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43349 } }
I20260810 18:08:54.579332 3626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:08:54.580626 3633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c1215894a714060a248ef8bea1c1355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1215894a714060a248ef8bea1c1355" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43349 } } }
I20260810 18:08:54.580772 3633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [sys.catalog]: This master's current role is: LEADER
I20260810 18:08:54.580989 3634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c1215894a714060a248ef8bea1c1355. Latest consensus state: current_term: 1 leader_uuid: "4c1215894a714060a248ef8bea1c1355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1215894a714060a248ef8bea1c1355" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43349 } } }
I20260810 18:08:54.581152 3634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [sys.catalog]: This master's current role is: LEADER
I20260810 18:08:54.581209 3643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:08:54.584218 3643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:08:54.584720 3626 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 18:08:54.594946 3643 catalog_manager.cc:1347] Generated new cluster ID: c4b11fdfa11245e98c0ab35f05da647d
I20260810 18:08:54.595041 3643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:08:54.622208 3643 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:08:54.623703 3643 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:08:54.630793 3643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355: Generated new TSK 0
I20260810 18:08:54.631731 3643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:08:55.321056 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.182.84:60865/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43349
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:08:55.460606 3682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:08:55.460930 3682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:08:55.461045 3682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:08:55.465260 3682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:08:55.465359 3682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:08:55.465466 3682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:08:55.471424 3682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.182.84:60865/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:08:55.472859 3682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:08:55.473996 3682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:08:55.485719 3689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:55.485719 3688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:55.486622 3691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:55.486622 3682 server_base.cc:1034] running on GCE node
I20260810 18:08:55.487142 3682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:08:55.488081 3682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:08:55.489295 3682 hybrid_clock.cc:648] HybridClock initialized: now 1786385335489259 us; error 57 us; skew 500 ppm
Aug 10 18:08:55 dist-test-slave-27sr krb5kdc[3055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385335, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:08:55.492851 3682 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:08:55.494220 3682 webserver.cc:460] Webserver started at http://127.21.182.65:36475/ using document root <none> and password file <none>
I20260810 18:08:55.494989 3682 fs_manager.cc:359] Metadata directory not provided
I20260810 18:08:55.495090 3682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:08:55.495376 3682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:08:55 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385335, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:55.513494 3682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1"
format_stamp: "Formatted at 2026-08-10 18:08:55 on dist-test-slave-27sr"
server_key: "3d869df63f5fb44734a4a549f08f5a30"
server_key_iv: "59397a959632d2e234345e964fb1107a"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:55.514271 3682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1"
format_stamp: "Formatted at 2026-08-10 18:08:55 on dist-test-slave-27sr"
server_key: "3d869df63f5fb44734a4a549f08f5a30"
server_key_iv: "59397a959632d2e234345e964fb1107a"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:55.519480 3682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 18:08:55.541919 3701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:55.543843 3682 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260810 18:08:55.544018 3682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1"
format_stamp: "Formatted at 2026-08-10 18:08:55 on dist-test-slave-27sr"
server_key: "3d869df63f5fb44734a4a549f08f5a30"
server_key_iv: "59397a959632d2e234345e964fb1107a"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:55.544169 3682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:08:55.563644 3682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:08:55.567608 3682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:08:55.567878 3682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:08:55.568790 3682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:08:55.569864 3682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:08:55.569916 3682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:55.569983 3682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:08:55.570016 3682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:55.582932 3682 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:34581
I20260810 18:08:55.584081 3682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:08:55.587316 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 3682
I20260810 18:08:55.587458 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:08:55.587764 22233 external_mini_cluster.cc:1428] Setting key 17acb7dc15759e6d1e8e8f63daa5701a
I20260810 18:08:55.596204 3826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:34581 every 8 connection(s)
Aug 10 18:08:55 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385335, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
I20260810 18:08:55.620366 3831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:55.601840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:48101 (local address 127.21.182.126:43349)
0810 18:08:55.606001 (+ 4161us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:55.606010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:55.606027 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:08:55.606075 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:55.606078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:55.606131 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:55.606220 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:08:55.608027 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:55.608995 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:55.610822 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:55.611192 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:55.611313 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:55.614964 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:08:55.614969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:08:55.614972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:08:55.615008 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:08:55.617185 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:55.617971 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:55.617976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:55.617978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:55.618081 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:55.618540 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:55.618544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:55.618546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:55.618842 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:08:55.618940 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:55.620010 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0810 18:08:55.620153 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4069,"thread_start_us":206,"threads_started":1}
I20260810 18:08:55.622332 3830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:55.600102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:43349 (local address 127.21.182.65:48101)
0810 18:08:55.604277 (+ 4175us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:55.604349 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:55.605504 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:55.606593 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:55.606603 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:55.607028 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:08:55.607771 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:55.607792 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:55.609164 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:55.609168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:55.610624 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:55.610637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:55.611524 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:55.611529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:55.611761 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:55.612629 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:08:55.612657 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:08:55.614683 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:08:55.617362 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:55.617370 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:55.617392 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:55.617833 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:55.618193 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:55.618197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:55.618199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:55.618425 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:55.618999 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:08:55.619007 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:08:55.619387 (+ 380us) client_negotiation.cc:769] Sending connection context
0810 18:08:55.619610 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 18:08:55.621011 (+ 1401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3898,"thread_start_us":493,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
I20260810 18:08:55.624323 3827 heartbeater.cc:347] Connected to a master server at 127.21.182.126:43349
I20260810 18:08:55.624609 3827 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:55.625295 3827 heartbeater.cc:511] Master 127.21.182.126:43349 requested a full tablet report, sending...
I20260810 18:08:55.627614 3585 ts_manager.cc:194] Registered new tserver with Master: 7c5c2ef022cd40e6a7fa1cf56ea390e1 (127.21.182.65:34581)
I20260810 18:08:55.629498 3585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:48101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:08:55.647868 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.182.84:60865/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43349
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:08:55 dist-test-slave-27sr krb5kdc[3055](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786385335, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 18:08:55.801887 3835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:08:55.802199 3835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:08:55.802310 3835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:08:55.806835 3835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:08:55.806926 3835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:08:55.807030 3835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:08:55.812939 3835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.182.84:60865/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:08:55.814781 3835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:08:55.816094 3835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:08:55.828469 3842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:55.828505 3843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:55.829020 3846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:55.830083 3835 server_base.cc:1034] running on GCE node
I20260810 18:08:55.830647 3835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:08:55.831604 3835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:08:55.832906 3835 hybrid_clock.cc:648] HybridClock initialized: now 1786385335832843 us; error 103 us; skew 500 ppm
Aug 10 18:08:55 dist-test-slave-27sr krb5kdc[3055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385335, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:08:55.837102 3835 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:08:55.838290 3835 webserver.cc:460] Webserver started at http://127.21.182.66:44721/ using document root <none> and password file <none>
I20260810 18:08:55.839010 3835 fs_manager.cc:359] Metadata directory not provided
I20260810 18:08:55.839082 3835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:08:55.839340 3835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:08:55 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385335, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:55.857798 3835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "a01311712a80480e9ac2b2bbee6b9ed2"
format_stamp: "Formatted at 2026-08-10 18:08:55 on dist-test-slave-27sr"
server_key: "876a65327efe07bfdf66bd95da6afeee"
server_key_iv: "839cc1edc18983632daaf9d390848ad7"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:55.858860 3835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "a01311712a80480e9ac2b2bbee6b9ed2"
format_stamp: "Formatted at 2026-08-10 18:08:55 on dist-test-slave-27sr"
server_key: "876a65327efe07bfdf66bd95da6afeee"
server_key_iv: "839cc1edc18983632daaf9d390848ad7"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:55.865963 3835 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260810 18:08:55.888964 3853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:55.891167 3835 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260810 18:08:55.891372 3835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "a01311712a80480e9ac2b2bbee6b9ed2"
format_stamp: "Formatted at 2026-08-10 18:08:55 on dist-test-slave-27sr"
server_key: "876a65327efe07bfdf66bd95da6afeee"
server_key_iv: "839cc1edc18983632daaf9d390848ad7"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:55.891561 3835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:08:55.922477 3835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:08:55.926764 3835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:08:55.927037 3835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:08:55.927990 3835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:08:55.929545 3835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:08:55.929657 3835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:55.929766 3835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:08:55.929837 3835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:55.942731 3835 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:43751
I20260810 18:08:55.944068 3835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:08:55.944330 3975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:43751 every 8 connection(s)
I20260810 18:08:55.948894 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 3835
I20260810 18:08:55.949038 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:08:55.949396 22233 external_mini_cluster.cc:1428] Setting key ad404f1854d42d95f54c97bff040d4c4
Aug 10 18:08:55 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385335, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:08:55.967171 3831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:55.950503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:45755 (local address 127.21.182.126:43349)
0810 18:08:55.950659 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:55.950670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:55.951871 (+ 1201us) server_negotiation.cc:407] Connection header received
0810 18:08:55.953355 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:55.953359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:55.953412 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:55.953462 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:08:55.955433 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:55.956310 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:55.958137 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:55.958502 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:55.958580 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:55.962086 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:08:55.962093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:08:55.962096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:08:55.962134 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:08:55.964163 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:55.965121 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:55.965125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:55.965127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:55.965226 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:55.965682 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:55.965686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:55.965688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:55.965947 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:08:55.966022 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:55.966825 (+ 803us) server_negotiation.cc:299] Negotiation successful
0810 18:08:55.966977 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
I20260810 18:08:55.969367 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:55.950434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:43349 (local address 127.21.182.66:45755)
0810 18:08:55.951699 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:55.951770 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:55.953091 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:55.953900 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:55.953909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:55.954397 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:08:55.955247 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:55.955269 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:55.956486 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:55.956490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:55.957984 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:55.957997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:55.958854 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:55.958858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:55.959082 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:55.959832 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:08:55.959854 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:08:55.961819 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:08:55.964537 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:55.964546 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:55.964568 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:55.964986 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:55.965356 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:55.965359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:55.965361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:55.965563 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:55.966112 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:08:55.966120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:08:55.966488 (+ 368us) client_negotiation.cc:769] Sending connection context
0810 18:08:55.967696 (+ 1208us) client_negotiation.cc:240] Negotiation successful
0810 18:08:55.968139 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"thread_start_us":170,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
I20260810 18:08:55.971045 3977 heartbeater.cc:347] Connected to a master server at 127.21.182.126:43349
I20260810 18:08:55.971314 3977 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:55.972095 3977 heartbeater.cc:511] Master 127.21.182.126:43349 requested a full tablet report, sending...
I20260810 18:08:55.973489 3585 ts_manager.cc:194] Registered new tserver with Master: a01311712a80480e9ac2b2bbee6b9ed2 (127.21.182.66:43751)
I20260810 18:08:55.974419 3585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:45755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:08:55.996281 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.182.84:60865/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43349
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:08:56.171653 3983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:08:56.172155 3983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:08:56.172231 3983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:08:56.179232 3983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:08:56.179338 3983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:08:56.179445 3983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:08:56.188784 3983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.182.84:60865/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:08:56.190330 3983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:08:56.191850 3983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:08:56.208959 3990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:56.209568 3989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:56.210021 3993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:56.216096 3983 server_base.cc:1034] running on GCE node
I20260810 18:08:56.216971 3983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:08:56.218019 3983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:08:56.221547 3983 hybrid_clock.cc:648] HybridClock initialized: now 1786385336221449 us; error 114 us; skew 500 ppm
Aug 10 18:08:56 dist-test-slave-27sr krb5kdc[3055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385336, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:08:56.226871 3983 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
Aug 10 18:08:56 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385335, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:56.229022 3983 webserver.cc:460] Webserver started at http://127.21.182.67:39211/ using document root <none> and password file <none>
I20260810 18:08:56.230007 3983 fs_manager.cc:359] Metadata directory not provided
I20260810 18:08:56.230170 3983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:08:56.230561 3983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 18:08:56 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385336, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:56.254457 3983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "ea6dbb32e8344530ae8bbdf194627b99"
format_stamp: "Formatted at 2026-08-10 18:08:56 on dist-test-slave-27sr"
server_key: "7b791f5054cabef6df1c11b7ec507520"
server_key_iv: "8d9e33f1a382e388f65f55999d3139d9"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:56.255875 3983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "ea6dbb32e8344530ae8bbdf194627b99"
format_stamp: "Formatted at 2026-08-10 18:08:56 on dist-test-slave-27sr"
server_key: "7b791f5054cabef6df1c11b7ec507520"
server_key_iv: "8d9e33f1a382e388f65f55999d3139d9"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:56.273870 3983 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.003s sys 0.002s
I20260810 18:08:56.299283 4000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:56.304749 3983 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260810 18:08:56.305020 3983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "ea6dbb32e8344530ae8bbdf194627b99"
format_stamp: "Formatted at 2026-08-10 18:08:56 on dist-test-slave-27sr"
server_key: "7b791f5054cabef6df1c11b7ec507520"
server_key_iv: "8d9e33f1a382e388f65f55999d3139d9"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:56.305277 3983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:08:56.330600 3983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:08:56.335420 3983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:08:56.335677 3983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:08:56.336513 3983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:08:56.337643 3983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:08:56.337723 3983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:56.337821 3983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:08:56.337879 3983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:56.352456 3983 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:40603
I20260810 18:08:56.352689 4125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:40603 every 8 connection(s)
I20260810 18:08:56.353842 3983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:08:56.363972 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 3983
I20260810 18:08:56.364133 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:08:56.364516 22233 external_mini_cluster.cc:1428] Setting key 5153357a7ee094dcf5363b9dc67a5f0a
Aug 10 18:08:56 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385336, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:08:56.373921 3831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:56.356417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:42307 (local address 127.21.182.126:43349)
0810 18:08:56.356565 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:56.356570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:56.357869 (+ 1299us) server_negotiation.cc:407] Connection header received
0810 18:08:56.359721 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:56.359726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:56.359793 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:56.359963 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:08:56.361931 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:56.362738 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:56.364887 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:56.365312 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:56.365402 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:56.369049 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:08:56.369053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:08:56.369057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:08:56.369091 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:08:56.371042 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:56.371799 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:56.371804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:56.371806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:56.371947 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:56.372416 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:56.372421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:56.372423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:56.372698 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:08:56.372834 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:56.373565 (+ 731us) server_negotiation.cc:299] Negotiation successful
0810 18:08:56.373731 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:08:56.375082 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:56.356937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:43349 (local address 127.21.182.67:42307)
0810 18:08:56.357663 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:56.357733 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:56.359391 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:56.360180 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:56.360191 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:56.360793 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:08:56.361700 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:56.361726 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:56.362917 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:56.362922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:56.364686 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:56.364702 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:56.365657 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:56.365661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:56.365910 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:56.366739 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:08:56.366765 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:08:56.368833 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:08:56.371203 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:56.371211 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:56.371233 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:56.371660 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:56.372053 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:56.372058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:56.372061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:56.372290 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:56.372834 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:08:56.372843 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:08:56.373232 (+ 389us) client_negotiation.cc:769] Sending connection context
0810 18:08:56.373475 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 18:08:56.373881 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":233,"threads_started":1}
I20260810 18:08:56.376947 4126 heartbeater.cc:347] Connected to a master server at 127.21.182.126:43349
I20260810 18:08:56.377256 4126 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:56.377935 4126 heartbeater.cc:511] Master 127.21.182.126:43349 requested a full tablet report, sending...
I20260810 18:08:56.381714 3585 ts_manager.cc:194] Registered new tserver with Master: ea6dbb32e8344530ae8bbdf194627b99 (127.21.182.67:40603)
I20260810 18:08:56.382786 3585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:42307
I20260810 18:08:56.384784 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:08:56.385725 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 3682
I20260810 18:08:56.400142 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 3835
I20260810 18:08:56.410655 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 3983
I20260810 18:08:56.422618 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 3553
I20260810 18:08:56.434509 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:43349
--webserver_interface=127.21.182.126
--webserver_port=33145
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:43349
--ranger_config_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.182.84:60865/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.182.84:60865/kms with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:08:56.562309 4129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:08:56.562601 4129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:08:56.562645 4129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:08:56.567428 4129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:08:56.567507 4129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:08:56.567528 4129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:08:56.567544 4129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:08:56.567613 4129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:08:56.573348 4129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.182.84:60865/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:43349
--ranger_config_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:43349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=33145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:08:56.574579 4129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:08:56.575733 4129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:08:56.584681 4134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:56.584676 4136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:56.584707 4138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:56.585186 4129 server_base.cc:1034] running on GCE node
I20260810 18:08:56.585790 4129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:08:56.586928 4129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:08:56.588140 4129 hybrid_clock.cc:648] HybridClock initialized: now 1786385336588125 us; error 26 us; skew 500 ppm
Aug 10 18:08:56 dist-test-slave-27sr krb5kdc[3055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385336, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:08:56.590960 4129 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:08:56.591972 4129 webserver.cc:460] Webserver started at http://127.21.182.126:33145/ using document root <none> and password file <none>
I20260810 18:08:56.592566 4129 fs_manager.cc:359] Metadata directory not provided
I20260810 18:08:56.592666 4129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 18:08:56 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385336, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:56.677593 4129 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:08:56.680524 4147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:56.682149 4129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:08:56.682291 4129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "4c1215894a714060a248ef8bea1c1355"
format_stamp: "Formatted at 2026-08-10 18:08:54 on dist-test-slave-27sr"
server_key: "b355d673c6ad64e8426fa33277f3384e"
server_key_iv: "43835f8f040a371f848e1a934fb8525d"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:56.683004 4129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:08:56.704666 4129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:08:56.707432 4129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:08:56.707603 4129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:08:56.715222 4129 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:43349
I20260810 18:08:56.715252 4203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:43349 every 8 connection(s)
I20260810 18:08:56.716389 4129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:08:56.720952 4204 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:08:56.722561 4204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355: Bootstrap starting.
I20260810 18:08:56.724431 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 4129
I20260810 18:08:56.724941 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.182.84:60865/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:34581
--local_ip_for_outbound_sockets=127.21.182.65
--tserver_master_addrs=127.21.182.126:43349
--webserver_port=36475
--webserver_interface=127.21.182.65
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.182.84:60865/kms with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:08:56.740105 4204 log.cc:824] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355: Log is configured to *not* fsync() on all Append() calls
I20260810 18:08:56.747133 4204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:08:56.747553 4204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355: Bootstrap complete.
I20260810 18:08:56.751852 4204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1215894a714060a248ef8bea1c1355" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43349 } }
I20260810 18:08:56.752307 4204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1215894a714060a248ef8bea1c1355, State: Initialized, Role: FOLLOWER
I20260810 18:08:56.752826 4204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1215894a714060a248ef8bea1c1355" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43349 } }
I20260810 18:08:56.752944 4204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:08:56.753034 4204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:08:56.753134 4204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:08:56.755034 4204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1215894a714060a248ef8bea1c1355" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43349 } }
I20260810 18:08:56.755401 4204 leader_election.cc:302] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c1215894a714060a248ef8bea1c1355; no voters:
I20260810 18:08:56.755786 4204 leader_election.cc:288] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:08:56.755851 4208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:08:56.756120 4208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [term 2 LEADER]: Becoming Leader. State: Replica: 4c1215894a714060a248ef8bea1c1355, State: Running, Role: LEADER
I20260810 18:08:56.756484 4208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1215894a714060a248ef8bea1c1355" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43349 } }
I20260810 18:08:56.757104 4204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:08:56.758014 4210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c1215894a714060a248ef8bea1c1355. Latest consensus state: current_term: 2 leader_uuid: "4c1215894a714060a248ef8bea1c1355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1215894a714060a248ef8bea1c1355" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43349 } } }
I20260810 18:08:56.758057 4209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4c1215894a714060a248ef8bea1c1355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1215894a714060a248ef8bea1c1355" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43349 } } }
I20260810 18:08:56.758147 4210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [sys.catalog]: This master's current role is: LEADER
I20260810 18:08:56.758178 4209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355 [sys.catalog]: This master's current role is: LEADER
I20260810 18:08:56.759647 4204 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 18:08:56.760002 4217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:08:56.766734 4217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:08:56.767654 4217 catalog_manager.cc:1259] Loaded cluster ID: c4b11fdfa11245e98c0ab35f05da647d
I20260810 18:08:56.767802 4217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:08:56.770475 4217 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:08:56.771275 4217 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355: Loaded TSK: 0
I20260810 18:08:56.772673 4217 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:08:56.884137 4206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:08:56.884443 4206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:08:56.884546 4206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:08:56.889217 4206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:08:56.889313 4206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:08:56.889421 4206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:08:56.895874 4206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.182.84:60865/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:34581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=36475
--tserver_master_addrs=127.21.182.126:43349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:08:56.897323 4206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:08:56.898556 4206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:08:56.912569 4245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:56.912945 4206 server_base.cc:1034] running on GCE node
W20260810 18:08:56.912703 4246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:56.913342 4248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:56.913838 4206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:08:56.914605 4206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:08:56.915962 4206 hybrid_clock.cc:648] HybridClock initialized: now 1786385336915868 us; error 48 us; skew 500 ppm
Aug 10 18:08:56 dist-test-slave-27sr krb5kdc[3055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385336, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:08:56.919327 4206 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:08:56.920715 4206 webserver.cc:460] Webserver started at http://127.21.182.65:36475/ using document root <none> and password file <none>
I20260810 18:08:56.921469 4206 fs_manager.cc:359] Metadata directory not provided
I20260810 18:08:56.921669 4206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 18:08:56 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385336, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:56.949229 4206 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:08:56.952804 4260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:56.954362 4206 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 18:08:56.954562 4206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1"
format_stamp: "Formatted at 2026-08-10 18:08:55 on dist-test-slave-27sr"
server_key: "3d869df63f5fb44734a4a549f08f5a30"
server_key_iv: "59397a959632d2e234345e964fb1107a"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:56.955273 4206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:08:56.980588 4206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:08:56.986088 4206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:08:56.986330 4206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:08:56.987120 4206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:08:56.988270 4206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:08:56.988346 4206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:56.988490 4206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:08:56.988547 4206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:57.003273 4206 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:34581
I20260810 18:08:57.004774 4206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:08:57.007758 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 4206
I20260810 18:08:57.008394 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.182.84:60865/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:43751
--local_ip_for_outbound_sockets=127.21.182.66
--tserver_master_addrs=127.21.182.126:43349
--webserver_port=44721
--webserver_interface=127.21.182.66
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.182.84:60865/kms with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:08:57.005522 4386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:34581 every 8 connection(s)
Aug 10 18:08:57 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385336, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:08:57.036556 4391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:57.008027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:43349 (local address 127.21.182.65:34611)
0810 18:08:57.015445 (+ 7418us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:57.015504 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:57.016633 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:57.018374 (+ 1741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:57.018384 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:57.018832 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:08:57.019579 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:57.019605 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:57.021954 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:57.021959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:57.023572 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:57.023588 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:57.025230 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:57.025236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:57.025469 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:57.026349 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:08:57.026377 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:08:57.028583 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:08:57.031880 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:57.031917 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:57.031936 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:57.032447 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:57.032974 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:57.032978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:57.032980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:57.033182 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:57.034181 (+ 999us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:08:57.034190 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:08:57.034570 (+ 380us) client_negotiation.cc:769] Sending connection context
0810 18:08:57.034826 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 18:08:57.035275 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7191,"thread_start_us":171,"threads_started":1}
I20260810 18:08:57.038686 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:57.007822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:34611 (local address 127.21.182.126:43349)
0810 18:08:57.015097 (+ 7275us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:57.015121 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:57.015677 (+ 556us) server_negotiation.cc:407] Connection header received
0810 18:08:57.017399 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:57.017411 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:57.017914 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:57.018296 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:08:57.019830 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:57.021742 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:57.023765 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:57.024713 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:57.024924 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:57.028932 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:08:57.028943 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:08:57.028971 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:08:57.029016 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:08:57.031683 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:57.032643 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:57.032652 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:57.032659 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:57.032838 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:57.033333 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:57.033337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:57.033339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:57.033947 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:08:57.034108 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:57.037012 (+ 2904us) server_negotiation.cc:299] Negotiation successful
0810 18:08:57.037538 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7052,"thread_start_us":139,"threads_started":1}
I20260810 18:08:57.042486 4387 heartbeater.cc:347] Connected to a master server at 127.21.182.126:43349
I20260810 18:08:57.042812 4387 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:57.043488 4387 heartbeater.cc:511] Master 127.21.182.126:43349 requested a full tablet report, sending...
W20260810 18:08:57.046211 4387 heartbeater.cc:644] Failed to heartbeat to 127.21.182.126:43349 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 18:08:57.046368 4387 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:57.046609 4387 heartbeater.cc:511] Master 127.21.182.126:43349 requested a full tablet report, sending...
I20260810 18:08:57.047811 4387 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:57.048064 4387 heartbeater.cc:511] Master 127.21.182.126:43349 requested a full tablet report, sending...
W20260810 18:08:57.049023 4387 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:08:57.180852 4392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:08:57.181195 4392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:08:57.181278 4392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:08:57.186983 4392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:08:57.187094 4392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:08:57.187189 4392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:08:57.193558 4392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.182.84:60865/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:43751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=44721
--tserver_master_addrs=127.21.182.126:43349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:08:57.194954 4392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:08:57.196187 4392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:08:57.206862 4399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:57.207420 4392 server_base.cc:1034] running on GCE node
W20260810 18:08:57.211421 4401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:57.212020 4398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:57.212729 4392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:08:57.213711 4392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:08:57.217023 4392 hybrid_clock.cc:648] HybridClock initialized: now 1786385337216991 us; error 42 us; skew 500 ppm
Aug 10 18:08:57 dist-test-slave-27sr krb5kdc[3055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385337, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:08:57.221546 4392 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:08:57.222951 4392 webserver.cc:460] Webserver started at http://127.21.182.66:44721/ using document root <none> and password file <none>
I20260810 18:08:57.223932 4392 fs_manager.cc:359] Metadata directory not provided
I20260810 18:08:57.224133 4392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 18:08:57 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385337, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:57.256844 4392 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.001s sys 0.003s
I20260810 18:08:57.260857 4410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:57.262702 4392 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 18:08:57.262868 4392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "a01311712a80480e9ac2b2bbee6b9ed2"
format_stamp: "Formatted at 2026-08-10 18:08:55 on dist-test-slave-27sr"
server_key: "876a65327efe07bfdf66bd95da6afeee"
server_key_iv: "839cc1edc18983632daaf9d390848ad7"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:57.263813 4392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:08:57.313323 4392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:08:57.317893 4392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:08:57.318147 4392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:08:57.319038 4392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:08:57.320317 4392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:08:57.320394 4392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:57.320458 4392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:08:57.320492 4392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:57.332948 4392 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:43751
I20260810 18:08:57.332967 4540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:43751 every 8 connection(s)
I20260810 18:08:57.334262 4392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:08:57.340096 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 4392
I20260810 18:08:57.340663 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.182.84:60865/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:40603
--local_ip_for_outbound_sockets=127.21.182.67
--tserver_master_addrs=127.21.182.126:43349
--webserver_port=39211
--webserver_interface=127.21.182.67
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.182.84:60865/kms with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:08:57 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385337, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:08:57.352149 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:57.336725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:38973 (local address 127.21.182.126:43349)
0810 18:08:57.336884 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:57.336888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:57.338102 (+ 1214us) server_negotiation.cc:407] Connection header received
0810 18:08:57.339128 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:57.339133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:57.339201 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:57.339312 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:08:57.340722 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:57.341267 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:57.342614 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:57.343020 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:57.343145 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:57.346673 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:08:57.346680 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:08:57.346687 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:08:57.346732 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:08:57.348734 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:57.349597 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:57.349603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:57.349608 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:57.349715 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:57.350349 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:57.350357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:57.350362 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:57.350668 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:08:57.350762 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:57.351693 (+ 931us) server_negotiation.cc:299] Negotiation successful
0810 18:08:57.351875 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:08:57.354394 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:57.337325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:43349 (local address 127.21.182.66:38973)
0810 18:08:57.337930 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:57.337978 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:57.338891 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:57.339513 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:57.339520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:57.339815 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:08:57.340549 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:57.340572 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:57.341425 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:57.341431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:57.342466 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:57.342477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:57.343381 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:57.343387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:57.343648 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:57.344377 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:08:57.344403 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:08:57.346397 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:08:57.348919 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:57.348929 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:57.348958 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:57.349445 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:57.349884 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:57.349889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:57.349894 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:57.350135 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:57.350864 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:08:57.350874 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:08:57.351316 (+ 442us) client_negotiation.cc:769] Sending connection context
0810 18:08:57.352685 (+ 1369us) client_negotiation.cc:240] Negotiation successful
0810 18:08:57.353144 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":177,"threads_started":1}
I20260810 18:08:57.355990 4541 heartbeater.cc:347] Connected to a master server at 127.21.182.126:43349
I20260810 18:08:57.356298 4541 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:57.356995 4541 heartbeater.cc:511] Master 127.21.182.126:43349 requested a full tablet report, sending...
W20260810 18:08:57.359221 4541 heartbeater.cc:644] Failed to heartbeat to 127.21.182.126:43349 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 18:08:57.359386 4541 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:57.359619 4541 heartbeater.cc:511] Master 127.21.182.126:43349 requested a full tablet report, sending...
I20260810 18:08:57.360845 4541 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:57.361063 4541 heartbeater.cc:511] Master 127.21.182.126:43349 requested a full tablet report, sending...
W20260810 18:08:57.361996 4541 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:08:57.500133 4545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:08:57.500463 4545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:08:57.500595 4545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:08:57.506043 4545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:08:57.506119 4545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:08:57.506183 4545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:08:57.514004 4545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.182.84:60865/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:40603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=39211
--tserver_master_addrs=127.21.182.126:43349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:08:57.515702 4545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:08:57.517073 4545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:08:57.530187 4551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:57.530402 4554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:08:57.531534 4552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:08:57.531949 4545 server_base.cc:1034] running on GCE node
I20260810 18:08:57.532552 4545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:08:57.533453 4545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:08:57.534860 4545 hybrid_clock.cc:648] HybridClock initialized: now 1786385337534826 us; error 50 us; skew 500 ppm
Aug 10 18:08:57 dist-test-slave-27sr krb5kdc[3055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385337, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:08:57.540167 4545 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:08:57.541554 4545 webserver.cc:460] Webserver started at http://127.21.182.67:39211/ using document root <none> and password file <none>
I20260810 18:08:57.542415 4545 fs_manager.cc:359] Metadata directory not provided
I20260810 18:08:57.542598 4545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 18:08:57 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385337, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:57.569859 4545 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.000s
I20260810 18:08:57.573681 4563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:57.575575 4545 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 18:08:57.575742 4545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "ea6dbb32e8344530ae8bbdf194627b99"
format_stamp: "Formatted at 2026-08-10 18:08:56 on dist-test-slave-27sr"
server_key: "7b791f5054cabef6df1c11b7ec507520"
server_key_iv: "8d9e33f1a382e388f65f55999d3139d9"
server_key_version: "kuduclusterkey@0"
I20260810 18:08:57.576594 4545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:08:57.611725 4545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:08:57.615409 4545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:08:57.615636 4545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:08:57.616470 4545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:08:57.617650 4545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:08:57.617725 4545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:57.617794 4545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:08:57.617838 4545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:08:57.630194 4545 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:40603
I20260810 18:08:57.631633 4701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:40603 every 8 connection(s)
I20260810 18:08:57.631837 4545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:08:57.641871 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 4545
Aug 10 18:08:57 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385337, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:08:57.651013 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:57.636116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:50889 (local address 127.21.182.126:43349)
0810 18:08:57.636284 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:57.636289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:57.636310 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:08:57.637002 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:57.637006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:57.637068 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:57.637179 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:08:57.638776 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:57.639632 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:57.641442 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:57.641794 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:57.641945 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:57.645289 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:08:57.645293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:08:57.645296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:08:57.645325 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:08:57.647353 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:57.648188 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:57.648193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:57.648196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:57.648306 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:57.649254 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:57.649257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:57.649259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:57.649525 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:08:57.649670 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:57.650653 (+ 983us) server_negotiation.cc:299] Negotiation successful
0810 18:08:57.650785 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 18:08:57.652237 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:57.634813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:43349 (local address 127.21.182.67:50889)
0810 18:08:57.635501 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:57.635564 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:57.636731 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:57.637411 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:57.637422 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:57.637817 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:08:57.638544 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:57.638565 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:57.639825 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:57.639828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:57.641262 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:57.641274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:57.642046 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:57.642051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:57.642267 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:57.643036 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:08:57.643061 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:08:57.645074 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:08:57.647490 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:57.647499 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:57.647523 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:57.648050 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:57.648826 (+ 776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:08:57.648833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:08:57.648838 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:08:57.649113 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:08:57.649688 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:08:57.649698 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:08:57.650124 (+ 426us) client_negotiation.cc:769] Sending connection context
0810 18:08:57.650350 (+ 226us) client_negotiation.cc:240] Negotiation successful
0810 18:08:57.650788 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":178,"threads_started":1}
I20260810 18:08:57.653650 4703 heartbeater.cc:347] Connected to a master server at 127.21.182.126:43349
I20260810 18:08:57.653847 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:57.643092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.21.182.126:43349)
0810 18:08:57.643458 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:57.643466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:57.643616 (+ 150us) server_negotiation.cc:407] Connection header received
0810 18:08:57.643770 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:57.643774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:57.643837 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:57.643998 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:08:57.644974 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:57.645746 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:57.647152 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:57.647526 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:57.647650 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:57.649398 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:08:57.649403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:08:57.649405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:08:57.649442 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:08:57.651225 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:57.651945 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:57.651949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:57.651950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:57.652061 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:57.652652 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:57.652658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:57.652660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:57.652911 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:08:57.653131 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:57.653354 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 18:08:57.653500 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":125,"threads_started":1}
I20260810 18:08:57.653964 4703 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:57.654647 4703 heartbeater.cc:511] Master 127.21.182.126:43349 requested a full tablet report, sending...
I20260810 18:08:57.656322 4166 ts_manager.cc:194] Registered new tserver with Master: ea6dbb32e8344530ae8bbdf194627b99 (127.21.182.67:40603)
I20260810 18:08:57.658016 4166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:50889
Aug 10 18:08:57 dist-test-slave-27sr krb5kdc[3055](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786385337, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:08:58.049368 4387 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:58.049734 4387 heartbeater.cc:511] Master 127.21.182.126:43349 requested a full tablet report, sending...
I20260810 18:08:58.051352 4165 ts_manager.cc:194] Registered new tserver with Master: 7c5c2ef022cd40e6a7fa1cf56ea390e1 (127.21.182.65:34581)
I20260810 18:08:58.052280 4165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:34611
I20260810 18:08:58.362337 4541 heartbeater.cc:464] Registering TS with master...
I20260810 18:08:58.362718 4541 heartbeater.cc:511] Master 127.21.182.126:43349 requested a full tablet report, sending...
I20260810 18:08:58.363804 4165 ts_manager.cc:194] Registered new tserver with Master: a01311712a80480e9ac2b2bbee6b9ed2 (127.21.182.66:43751)
I20260810 18:08:58.364765 4165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:38973
I20260810 18:08:58.366814 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:08:58.377950 4720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.368719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.21.182.126:43349)
0810 18:08:58.369073 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:58.369080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:58.369151 (+ 71us) server_negotiation.cc:407] Connection header received
0810 18:08:58.369348 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:58.369351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:58.369408 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:58.369511 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:08:58.370435 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.371055 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.371932 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.372248 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.372360 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.374279 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:08:58.374283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:08:58.374285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:08:58.374314 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:08:58.376080 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:58.376563 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:58.376572 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:58.376573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:58.376664 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:08:58.376966 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:08:58.376970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:08:58.376972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:08:58.377181 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:08:58.377341 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:58.377594 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 18:08:58.377736 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":95,"threads_started":1}
I20260810 18:08:58.382462 4165 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:08:58.388092 4165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:08:58.422259 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.410904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:43751 (local address 127.0.0.1:55416)
0810 18:08:58.415361 (+ 4457us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:58.415439 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:58.415624 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:58.416490 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:58.416498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:58.416522 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:08:58.416671 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:58.416678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.418226 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.418230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:58.420425 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:58.420437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.421643 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.421647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:58.421778 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.421830 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 18:08:58.421930 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 18:08:58.422066 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4271,"mutex_wait_us":1,"thread_start_us":131,"threads_started":1}
I20260810 18:08:58.423386 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.411743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.21.182.66:43751)
0810 18:08:58.415081 (+ 3338us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:58.415089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:58.415739 (+ 650us) server_negotiation.cc:407] Connection header received
0810 18:08:58.415845 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:58.415851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:58.416139 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:58.416306 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:08:58.416852 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.418085 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.420611 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.421485 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.422880 (+ 1395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.423050 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:58.423112 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 18:08:58.423235 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3201,"thread_start_us":80,"threads_started":1}
I20260810 18:08:58.423496 4729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.412834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:40603 (local address 127.0.0.1:53452)
0810 18:08:58.415874 (+ 3040us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:58.416160 (+ 286us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:58.416268 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:58.417068 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:58.417072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:58.417091 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:08:58.417182 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:58.417189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.418629 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.418632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:58.421409 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:58.421415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.423056 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.423061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:58.423164 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.423192 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:08:58.423263 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 18:08:58.423344 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2975,"mutex_wait_us":1294,"thread_start_us":115,"threads_started":1}
I20260810 18:08:58.425410 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.413339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53452 (local address 127.21.182.67:40603)
0810 18:08:58.416119 (+ 2780us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:58.416126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:58.416373 (+ 247us) server_negotiation.cc:407] Connection header received
0810 18:08:58.416466 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:58.416474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:58.416706 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:58.416878 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:08:58.417338 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.418477 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.421543 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.422450 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.422575 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.423994 (+ 1419us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:58.424064 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 18:08:58.425249 (+ 1185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2659,"thread_start_us":124,"threads_started":1}
Aug 10 18:08:58 dist-test-slave-27sr krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385337, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for HTTP/127.21.182.84@KRBTEST.COM
I20260810 18:08:58.430956 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.410778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:34581 (local address 127.0.0.1:55832)
0810 18:08:58.415563 (+ 4785us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:58.415577 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:58.415652 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:58.421144 (+ 5492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:58.421147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:58.421169 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:08:58.421265 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:58.421272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.427753 (+ 6481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.427758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:58.429033 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:58.429049 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.430485 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.430489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:58.430596 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.430623 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:08:58.430703 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 18:08:58.430793 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2387,"thread_start_us":147,"threads_started":1}
I20260810 18:08:58.430460 4625 tablet_service.cc:1467] Processing CreateTablet for tablet 065a643161b84b2395b04eb0da120bfb (DEFAULT_TABLE table=test-table [id=a6f1884a86cf45379a709c0e6f52e333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:08:58.432428 4625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065a643161b84b2395b04eb0da120bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:08:58.432729 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.411732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.21.182.65:34581)
0810 18:08:58.420317 (+ 8585us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:58.420325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:58.420355 (+ 30us) server_negotiation.cc:407] Connection header received
0810 18:08:58.420468 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:58.420475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:58.420754 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:58.420954 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:08:58.426337 (+ 5383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.427586 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.429226 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.430104 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.430239 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.430405 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:58.432402 (+ 1997us) server_negotiation.cc:299] Negotiation successful
0810 18:08:58.432537 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8441,"thread_start_us":75,"threads_started":1}
I20260810 18:08:58.427429 4446 tablet_service.cc:1467] Processing CreateTablet for tablet 065a643161b84b2395b04eb0da120bfb (DEFAULT_TABLE table=test-table [id=a6f1884a86cf45379a709c0e6f52e333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:08:58.436455 4446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065a643161b84b2395b04eb0da120bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:08:58.436338 4314 tablet_service.cc:1467] Processing CreateTablet for tablet 065a643161b84b2395b04eb0da120bfb (DEFAULT_TABLE table=test-table [id=a6f1884a86cf45379a709c0e6f52e333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:08:58.437552 4314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065a643161b84b2395b04eb0da120bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:08:58.456360 4732 tablet_bootstrap.cc:492] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99: Bootstrap starting.
I20260810 18:08:58.459589 4732 tablet_bootstrap.cc:654] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99: Neither blocks nor log segments found. Creating new log.
I20260810 18:08:58.460943 4732 log.cc:824] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99: Log is configured to *not* fsync() on all Append() calls
I20260810 18:08:58.462218 4734 tablet_bootstrap.cc:492] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1: Bootstrap starting.
I20260810 18:08:58.463162 4731 tablet_bootstrap.cc:492] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2: Bootstrap starting.
I20260810 18:08:58.465914 4731 tablet_bootstrap.cc:654] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2: Neither blocks nor log segments found. Creating new log.
I20260810 18:08:58.466784 4734 tablet_bootstrap.cc:654] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1: Neither blocks nor log segments found. Creating new log.
I20260810 18:08:58.466905 4731 log.cc:824] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2: Log is configured to *not* fsync() on all Append() calls
I20260810 18:08:58.468017 4734 log.cc:824] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1: Log is configured to *not* fsync() on all Append() calls
I20260810 18:08:58.470007 4731 tablet_bootstrap.cc:492] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2: No bootstrap required, opened a new log
I20260810 18:08:58.470256 4731 ts_tablet_manager.cc:1397] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:08:58.474524 4731 raft_consensus.cc:348] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34581 } } peers { permanent_uuid: "a01311712a80480e9ac2b2bbee6b9ed2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43751 } } peers { permanent_uuid: "ea6dbb32e8344530ae8bbdf194627b99" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40603 } }
I20260810 18:08:58.474874 4731 raft_consensus.cc:374] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:08:58.474928 4731 raft_consensus.cc:729] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a01311712a80480e9ac2b2bbee6b9ed2, State: Initialized, Role: FOLLOWER
I20260810 18:08:58.475502 4731 consensus_queue.cc:260] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34581 } } peers { permanent_uuid: "a01311712a80480e9ac2b2bbee6b9ed2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43751 } } peers { permanent_uuid: "ea6dbb32e8344530ae8bbdf194627b99" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40603 } }
I20260810 18:08:58.476897 4731 ts_tablet_manager.cc:1428] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:08:58.477355 4541 heartbeater.cc:503] Master 127.21.182.126:43349 was elected leader, sending a full tablet report...
I20260810 18:08:58.478003 4732 tablet_bootstrap.cc:492] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99: No bootstrap required, opened a new log
I20260810 18:08:58.478236 4732 ts_tablet_manager.cc:1397] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.004s
I20260810 18:08:58.484270 4732 raft_consensus.cc:348] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34581 } } peers { permanent_uuid: "a01311712a80480e9ac2b2bbee6b9ed2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43751 } } peers { permanent_uuid: "ea6dbb32e8344530ae8bbdf194627b99" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40603 } }
I20260810 18:08:58.484723 4732 raft_consensus.cc:374] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:08:58.484848 4732 raft_consensus.cc:729] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6dbb32e8344530ae8bbdf194627b99, State: Initialized, Role: FOLLOWER
I20260810 18:08:58.485534 4732 consensus_queue.cc:260] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34581 } } peers { permanent_uuid: "a01311712a80480e9ac2b2bbee6b9ed2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43751 } } peers { permanent_uuid: "ea6dbb32e8344530ae8bbdf194627b99" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40603 } }
I20260810 18:08:58.486336 4703 heartbeater.cc:503] Master 127.21.182.126:43349 was elected leader, sending a full tablet report...
I20260810 18:08:58.489835 4734 tablet_bootstrap.cc:492] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1: No bootstrap required, opened a new log
I20260810 18:08:58.490243 4734 ts_tablet_manager.cc:1397] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20260810 18:08:58.490978 4732 ts_tablet_manager.cc:1428] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99: Time spent starting tablet: real 0.013s user 0.007s sys 0.001s
I20260810 18:08:58.496222 4734 raft_consensus.cc:348] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34581 } } peers { permanent_uuid: "a01311712a80480e9ac2b2bbee6b9ed2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43751 } } peers { permanent_uuid: "ea6dbb32e8344530ae8bbdf194627b99" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40603 } }
I20260810 18:08:58.496714 4734 raft_consensus.cc:374] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:08:58.496819 4734 raft_consensus.cc:729] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c5c2ef022cd40e6a7fa1cf56ea390e1, State: Initialized, Role: FOLLOWER
I20260810 18:08:58.497391 4734 consensus_queue.cc:260] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34581 } } peers { permanent_uuid: "a01311712a80480e9ac2b2bbee6b9ed2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43751 } } peers { permanent_uuid: "ea6dbb32e8344530ae8bbdf194627b99" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40603 } }
I20260810 18:08:58.498217 4387 heartbeater.cc:503] Master 127.21.182.126:43349 was elected leader, sending a full tablet report...
I20260810 18:08:58.498715 4734 ts_tablet_manager.cc:1428] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 18:08:58.507591 4739 raft_consensus.cc:482] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:08:58.507833 4739 raft_consensus.cc:504] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34581 } } peers { permanent_uuid: "a01311712a80480e9ac2b2bbee6b9ed2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43751 } } peers { permanent_uuid: "ea6dbb32e8344530ae8bbdf194627b99" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40603 } }
W20260810 18:08:58.508399 4389 tablet.cc:2369] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:08:58.509330 4739 leader_election.cc:288] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a01311712a80480e9ac2b2bbee6b9ed2 (127.21.182.66:43751), ea6dbb32e8344530ae8bbdf194627b99 (127.21.182.67:40603)
I20260810 18:08:58.512253 4738 raft_consensus.cc:482] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:08:58.512526 4738 raft_consensus.cc:504] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34581 } } peers { permanent_uuid: "a01311712a80480e9ac2b2bbee6b9ed2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43751 } } peers { permanent_uuid: "ea6dbb32e8344530ae8bbdf194627b99" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40603 } }
I20260810 18:08:58.513973 4738 leader_election.cc:288] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c5c2ef022cd40e6a7fa1cf56ea390e1 (127.21.182.65:34581), a01311712a80480e9ac2b2bbee6b9ed2 (127.21.182.66:43751)
I20260810 18:08:58.514899 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.510055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:59731 (local address 127.21.182.66:43751)
0810 18:08:58.510191 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:58.510195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:58.510206 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:08:58.510254 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:58.510257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:58.510297 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:58.510401 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:08:58.510764 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.511533 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.513455 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.514036 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.514145 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.514210 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:58.514633 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 18:08:58.514727 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:08:58.515450 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.509574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:43751 (local address 127.21.182.65:59731)
0810 18:08:58.509900 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:58.509918 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:58.510039 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:58.510405 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:58.510408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:58.510428 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:08:58.510601 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:58.510608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.512024 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.512027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:58.513297 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:58.513308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.514320 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.514323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:58.514445 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.514478 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 18:08:58.515210 (+ 732us) client_negotiation.cc:240] Negotiation successful
0810 18:08:58.515291 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":145,"threads_started":1}
I20260810 18:08:58.516371 4485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065a643161b84b2395b04eb0da120bfb" candidate_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a01311712a80480e9ac2b2bbee6b9ed2" is_pre_election: true
I20260810 18:08:58.516783 4485 raft_consensus.cc:2454] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c5c2ef022cd40e6a7fa1cf56ea390e1 in term 0.
I20260810 18:08:58.517563 4261 leader_election.cc:302] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c5c2ef022cd40e6a7fa1cf56ea390e1, a01311712a80480e9ac2b2bbee6b9ed2; no voters:
I20260810 18:08:58.518108 4739 raft_consensus.cc:2781] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:08:58.518217 4739 raft_consensus.cc:482] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:08:58.518265 4739 raft_consensus.cc:3037] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:08:58.519847 4739 raft_consensus.cc:504] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34581 } } peers { permanent_uuid: "a01311712a80480e9ac2b2bbee6b9ed2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43751 } } peers { permanent_uuid: "ea6dbb32e8344530ae8bbdf194627b99" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40603 } }
I20260810 18:08:58.520629 4485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065a643161b84b2395b04eb0da120bfb" candidate_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a01311712a80480e9ac2b2bbee6b9ed2"
I20260810 18:08:58.520782 4485 raft_consensus.cc:3037] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:08:58.522468 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.515533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:36987 (local address 127.21.182.65:34581)
0810 18:08:58.515786 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:58.515791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:58.516501 (+ 710us) server_negotiation.cc:407] Connection header received
0810 18:08:58.516674 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:58.516677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:58.516716 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:58.516757 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:08:58.518035 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.518793 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.520890 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.521483 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.521598 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.521651 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:58.522207 (+ 556us) server_negotiation.cc:299] Negotiation successful
0810 18:08:58.522299 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260810 18:08:58.522558 4485 raft_consensus.cc:2454] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c5c2ef022cd40e6a7fa1cf56ea390e1 in term 1.
I20260810 18:08:58.522998 4744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.514374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:34581 (local address 127.21.182.67:36987)
0810 18:08:58.516414 (+ 2040us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:58.516429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:58.516570 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:58.516865 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:58.516868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:58.516881 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:08:58.517213 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:58.517224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.519250 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.519253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:58.520566 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:58.520577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.521791 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.521794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:58.521962 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.522001 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 18:08:58.522713 (+ 712us) client_negotiation.cc:240] Negotiation successful
0810 18:08:58.522837 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1935,"thread_start_us":165,"threads_started":1}
I20260810 18:08:58.523758 4332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065a643161b84b2395b04eb0da120bfb" candidate_uuid: "ea6dbb32e8344530ae8bbdf194627b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" is_pre_election: true
I20260810 18:08:58.524102 4332 raft_consensus.cc:2379] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6dbb32e8344530ae8bbdf194627b99 in current term 1: Already voted for candidate 7c5c2ef022cd40e6a7fa1cf56ea390e1 in this term.
I20260810 18:08:58.524318 4261 leader_election.cc:302] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c5c2ef022cd40e6a7fa1cf56ea390e1, a01311712a80480e9ac2b2bbee6b9ed2; no voters:
I20260810 18:08:58.524698 4739 leader_election.cc:288] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [CANDIDATE]: Term 1 election: Requested vote from peers a01311712a80480e9ac2b2bbee6b9ed2 (127.21.182.66:43751), ea6dbb32e8344530ae8bbdf194627b99 (127.21.182.67:40603)
I20260810 18:08:58.524796 4739 raft_consensus.cc:2781] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:08:58.524861 4739 raft_consensus.cc:686] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [term 1 LEADER]: Becoming Leader. State: Replica: 7c5c2ef022cd40e6a7fa1cf56ea390e1, State: Running, Role: LEADER
I20260810 18:08:58.525233 4739 consensus_queue.cc:237] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34581 } } peers { permanent_uuid: "a01311712a80480e9ac2b2bbee6b9ed2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43751 } } peers { permanent_uuid: "ea6dbb32e8344530ae8bbdf194627b99" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40603 } }
I20260810 18:08:58.528878 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.515651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:52603 (local address 127.21.182.66:43751)
0810 18:08:58.515873 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:58.515877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:58.517585 (+ 1708us) server_negotiation.cc:407] Connection header received
0810 18:08:58.517783 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:58.517786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:58.517836 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:58.517883 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:08:58.520711 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.523443 (+ 2732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.527243 (+ 3800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.527877 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.528027 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.528097 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:58.528633 (+ 536us) server_negotiation.cc:299] Negotiation successful
0810 18:08:58.528740 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260810 18:08:58.529477 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.514868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:43751 (local address 127.21.182.67:52603)
0810 18:08:58.517479 (+ 2611us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:58.517493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:58.517654 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:58.518947 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:58.518951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:58.518971 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:08:58.519110 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:58.519116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.525203 (+ 6087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.525206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:58.526441 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:58.526453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.528246 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.528258 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:58.528409 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.528438 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 18:08:58.529225 (+ 787us) client_negotiation.cc:240] Negotiation successful
0810 18:08:58.529312 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2548,"thread_start_us":171,"threads_started":1}
I20260810 18:08:58.530596 4485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065a643161b84b2395b04eb0da120bfb" candidate_uuid: "ea6dbb32e8344530ae8bbdf194627b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a01311712a80480e9ac2b2bbee6b9ed2" is_pre_election: true
I20260810 18:08:58.530823 4485 raft_consensus.cc:2379] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea6dbb32e8344530ae8bbdf194627b99 in current term 1: Already voted for candidate 7c5c2ef022cd40e6a7fa1cf56ea390e1 in this term.
I20260810 18:08:58.531591 4571 leader_election.cc:302] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea6dbb32e8344530ae8bbdf194627b99; no voters: 7c5c2ef022cd40e6a7fa1cf56ea390e1, a01311712a80480e9ac2b2bbee6b9ed2
I20260810 18:08:58.531553 4165 catalog_manager.cc:5495] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c5c2ef022cd40e6a7fa1cf56ea390e1 (127.21.182.65). New cstate: current_term: 1 leader_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a01311712a80480e9ac2b2bbee6b9ed2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea6dbb32e8344530ae8bbdf194627b99" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40603 } health_report { overall_health: UNKNOWN } } }
I20260810 18:08:58.532404 4738 raft_consensus.cc:3037] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:08:58.533886 4738 raft_consensus.cc:2726] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:08:58.535847 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.511709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:50609 (local address 127.21.182.67:40603)
0810 18:08:58.511847 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:58.511852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:58.511866 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:08:58.511950 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:58.511956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:58.512014 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:58.512086 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:08:58.516261 (+ 4175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.517028 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.534419 (+ 17391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.535061 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.535182 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.535252 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:58.535633 (+ 381us) server_negotiation.cc:299] Negotiation successful
0810 18:08:58.535721 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:08:58.535868 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.510880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:40603 (local address 127.21.182.65:50609)
0810 18:08:58.511305 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:58.511319 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:58.511413 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:58.515956 (+ 4543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:08:58.515960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:08:58.515986 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:08:58.516125 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:58.516131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.528104 (+ 11973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.528110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:58.529518 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:08:58.529535 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.535373 (+ 5838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.535377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:08:58.535495 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.535519 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:08:58.535611 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 18:08:58.535708 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":106,"threads_started":1}
I20260810 18:08:58.536788 4649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065a643161b84b2395b04eb0da120bfb" candidate_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6dbb32e8344530ae8bbdf194627b99" is_pre_election: true
I20260810 18:08:58.537056 4649 raft_consensus.cc:2454] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c5c2ef022cd40e6a7fa1cf56ea390e1 in term 1.
I20260810 18:08:58.537590 4648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065a643161b84b2395b04eb0da120bfb" candidate_uuid: "7c5c2ef022cd40e6a7fa1cf56ea390e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6dbb32e8344530ae8bbdf194627b99"
I20260810 18:08:58.539044 4648 raft_consensus.cc:2454] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c5c2ef022cd40e6a7fa1cf56ea390e1 in term 1.
I20260810 18:08:58.568754 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.563542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55836 (local address 127.21.182.65:34581)
0810 18:08:58.563723 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 18:08:58.563728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:08:58.563746 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:08:58.563852 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:08:58.563856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:08:58.563973 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:08:58.564103 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:08:58.564419 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.565276 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.566192 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:08:58.566603 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:08:58.566765 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:08:58.567179 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:08:58.568094 (+ 915us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:08:58.568258 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 18:08:58.568363 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 18:08:58.568476 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 18:08:58.578469 4648 raft_consensus.cc:1261] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [term 1 FOLLOWER]: Refusing update from remote peer 7c5c2ef022cd40e6a7fa1cf56ea390e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:08:58.578495 4485 raft_consensus.cc:1261] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2 [term 1 FOLLOWER]: Refusing update from remote peer 7c5c2ef022cd40e6a7fa1cf56ea390e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:08:58.579334 4739 consensus_queue.cc:1035] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6dbb32e8344530ae8bbdf194627b99" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 40603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:08:58.580255 4739 consensus_queue.cc:1035] T 065a643161b84b2395b04eb0da120bfb P 7c5c2ef022cd40e6a7fa1cf56ea390e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a01311712a80480e9ac2b2bbee6b9ed2" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:08:58.586259 4542 tablet.cc:2369] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:08:58.589480 4750 mvcc.cc:205] Tried to move back new op lower bound from 7317034346807812096 to 7317034346605809664. Current Snapshot: MvccSnapshot[applied={T|T < 7317034346807812096}]
I20260810 18:08:58.598079 4752 mvcc.cc:205] Tried to move back new op lower bound from 7317034346807812096 to 7317034346605809664. Current Snapshot: MvccSnapshot[applied={T|T < 7317034346807812096}]
I20260810 18:08:58.610213 4165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:08:58.610419 4165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:08:58.613564 4165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c1215894a714060a248ef8bea1c1355: Sending DeleteTablet for 3 replicas of tablet 065a643161b84b2395b04eb0da120bfb
I20260810 18:08:58.615491 4314 tablet_service.cc:1514] Processing DeleteTablet for tablet 065a643161b84b2395b04eb0da120bfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:08:58 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:55832
I20260810 18:08:58.616465 4625 tablet_service.cc:1514] Processing DeleteTablet for tablet 065a643161b84b2395b04eb0da120bfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:08:58 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:53452
I20260810 18:08:58.617000 4446 tablet_service.cc:1514] Processing DeleteTablet for tablet 065a643161b84b2395b04eb0da120bfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:08:58 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:55416
I20260810 18:08:58.618096 4761 tablet_replica.cc:330] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99: stopping tablet replica
I20260810 18:08:58.618393 4762 tablet_replica.cc:330] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2: stopping tablet replica
I20260810 18:08:58.618549 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 4206
I20260810 18:08:58.619951 4761 raft_consensus.cc:2229] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:08:58.620365 4761 raft_consensus.cc:2258] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:08:58.620647 4762 raft_consensus.cc:2229] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:08:58.620982 4762 raft_consensus.cc:2258] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:08:58.623224 4762 ts_tablet_manager.cc:1905] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:08:58.624897 4761 ts_tablet_manager.cc:1905] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:08:58.627951 4762 ts_tablet_manager.cc:1918] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:08:58.631039 4761 ts_tablet_manager.cc:1918] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:08:58.631254 4762 log.cc:1196] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/065a643161b84b2395b04eb0da120bfb
I20260810 18:08:58.631421 4761 log.cc:1196] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/065a643161b84b2395b04eb0da120bfb
I20260810 18:08:58.631879 4761 ts_tablet_manager.cc:1939] T 065a643161b84b2395b04eb0da120bfb P ea6dbb32e8344530ae8bbdf194627b99: Deleting consensus metadata
I20260810 18:08:58.632392 4762 ts_tablet_manager.cc:1939] T 065a643161b84b2395b04eb0da120bfb P a01311712a80480e9ac2b2bbee6b9ed2: Deleting consensus metadata
W20260810 18:08:58.632675 4151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:34581 (error 108)
I20260810 18:08:58.634012 4151 catalog_manager.cc:4841] TS ea6dbb32e8344530ae8bbdf194627b99 (127.21.182.67:40603): tablet 065a643161b84b2395b04eb0da120bfb (table test-table [id=a6f1884a86cf45379a709c0e6f52e333]) successfully deleted
I20260810 18:08:58.634123 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 4392
I20260810 18:08:58.636238 4729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:08:58.634382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:34581 (local address 127.0.0.1:55840)
0810 18:08:58.636042 (+ 1660us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:58.636129 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:34581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1547}
W20260810 18:08:58.637430 4151 catalog_manager.cc:4584] TS 7c5c2ef022cd40e6a7fa1cf56ea390e1 (127.21.182.65:34581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 065a643161b84b2395b04eb0da120bfb: Network error: Client connection negotiation failed: client connection to 127.21.182.65:34581: connect: Connection refused (error 111)
W20260810 18:08:58.648809 4729 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:08:58.647935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:43751 (local address 127.0.0.1:55432)
0810 18:08:58.648153 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 18:08:58.648185 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 18:08:58.648297 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:08:58.648659 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.66:43751: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":129}
I20260810 18:08:58.649708 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 4545
I20260810 18:08:58.662304 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 4129
I20260810 18:08:58.675174 22233 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 18:08:59.020881 22233 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 18:08:59.021054 22233 mini_ranger.cc:66] Stopping Ranger...
I20260810 18:08:59.366487 22233 mini_ranger.cc:68] Stopped Ranger
2026-08-10 18:08:59.366 UTC [3089] LOG: received smart shutdown request
2026-08-10 18:08:59.371 UTC [3089] LOG: background worker "logical replication launcher" (PID 3100) exited with exit code 1
2026-08-10 18:08:59.372 UTC [3094] LOG: shutting down
2026-08-10 18:08:59.533 UTC [3089] LOG: database system is shut down
2026-08-10T18:08:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53277 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:08:59 dist-test-slave-27sr krb5kdc[4772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:08:59 dist-test-slave-27sr krb5kdc[4772](info): set up 2 sockets
Aug 10 18:08:59 dist-test-slave-27sr krb5kdc[4772](info): commencing operation
krb5kdc: starting...
W20260810 18:09:00.807487 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:09:00 dist-test-slave-27sr krb5kdc[4772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:09:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:09:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:09:00.954643 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:41415
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:41415,127.21.182.125:43387,127.21.182.124:33679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:09:01.096537 4787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:01.096911 4787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:01.097024 4787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:01.101202 4787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:09:01.101315 4787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:01.101356 4787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:01.101399 4787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:09:01.101435 4787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:09:01.107012 4787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:41415,127.21.182.125:43387,127.21.182.124:33679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:41415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:01.108312 4787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:01.109433 4787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:01.118826 4796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:01.118834 4793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:01.119071 4787 server_base.cc:1034] running on GCE node
W20260810 18:09:01.118965 4794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:01.119856 4787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:01.121088 4787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:01.122287 4787 hybrid_clock.cc:648] HybridClock initialized: now 1786385341122271 us; error 32 us; skew 500 ppm
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385341, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:01.125294 4787 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:09:01.126271 4787 webserver.cc:460] Webserver started at http://127.21.182.126:39863/ using document root <none> and password file <none>
I20260810 18:09:01.126878 4787 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:01.126974 4787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:01.127233 4787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:09:01.129498 4787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "c8e83f42a0f44219be9ab6f22b02c98c"
format_stamp: "Formatted at 2026-08-10 18:09:01 on dist-test-slave-27sr"
server_key: "c6e1eaf37538969a8a7aa3e023f11e29"
server_key_iv: "9fbfb9b63914dc8875172e7e778465f4"
server_key_version: "encryptionkey@0"
I20260810 18:09:01.130098 4787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "c8e83f42a0f44219be9ab6f22b02c98c"
format_stamp: "Formatted at 2026-08-10 18:09:01 on dist-test-slave-27sr"
server_key: "c6e1eaf37538969a8a7aa3e023f11e29"
server_key_iv: "9fbfb9b63914dc8875172e7e778465f4"
server_key_version: "encryptionkey@0"
I20260810 18:09:01.134001 4787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:09:01.136574 4803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:01.137852 4787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:09:01.138010 4787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "c8e83f42a0f44219be9ab6f22b02c98c"
format_stamp: "Formatted at 2026-08-10 18:09:01 on dist-test-slave-27sr"
server_key: "c6e1eaf37538969a8a7aa3e023f11e29"
server_key_iv: "9fbfb9b63914dc8875172e7e778465f4"
server_key_version: "encryptionkey@0"
I20260810 18:09:01.138127 4787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:01.163866 4787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:01.167346 4787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:01.167595 4787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:01.175745 4787 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:41415
I20260810 18:09:01.176097 4859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:41415 every 8 connection(s)
I20260810 18:09:01.176908 4787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:09:01.180160 4860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:01.183862 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 4787
I20260810 18:09:01.184068 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:09:01.184405 22233 external_mini_cluster.cc:1428] Setting key eccbc0d95f12bcb0a05089ca09db3403
I20260810 18:09:01.184501 4860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.182.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.125@KRBTEST.COM" created.
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385341, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:01.204962 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.187298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:41415 (local address 127.0.0.1:41074)
0810 18:09:01.187819 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.187885 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:01.188764 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:01.189544 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:01.189550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:01.190030 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:01.190287 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.190301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.191554 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.191557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.192409 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.192419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.193084 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.193088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.193211 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.193879 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:01.193896 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:01.198931 (+ 5035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:01.201397 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.201409 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.201421 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.201688 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.202092 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.202096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.202098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.202236 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.202941 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:01.202948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:01.203101 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 18:09:01.203295 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 18:09:01.203828 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":202,"thread_start_us":157,"threads_started":1}
I20260810 18:09:01.204962 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.187299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.21.182.126:41415)
0810 18:09:01.187914 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.187929 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.188009 (+ 80us) server_negotiation.cc:407] Connection header received
0810 18:09:01.189068 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.189073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.189306 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.189528 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.190450 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.191423 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.192537 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.192915 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.193096 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.199122 (+ 6026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.199128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.199140 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.199188 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.201260 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.201825 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.201832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.201836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.201959 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.202350 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.202354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.202356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.202800 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.202961 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.203388 (+ 427us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.203769 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":450,"thread_start_us":151,"threads_started":1}
I20260810 18:09:01.208099 4860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:09:01.209149 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.208684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:43387 (local address 127.0.0.1:60094)
0810 18:09:01.208928 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.209049 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260810 18:09:01.209370 4806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.182.125:43387: connect: Connection refused (error 111)
I20260810 18:09:01.210331 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.210102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:43387 (local address 127.0.0.1:60106)
0810 18:09:01.210216 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.210272 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 18:09:01.210516 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.125:43387: Network error: Client connection negotiation failed: client connection to 127.21.182.125:43387: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.125.keytab.
Entry for principal kudu/127.21.182.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.125.keytab.
I20260810 18:09:01.216071 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.125:43387
--webserver_interface=127.21.182.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.125.keytab
--principal=kudu/127.21.182.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:41415,127.21.182.125:43387,127.21.182.124:33679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:09:01.259845 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } attempt: 1
I20260810 18:09:01.260991 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.260655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:43387 (local address 127.0.0.1:60110)
0810 18:09:01.260845 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.260910 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 18:09:01.261924 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.261685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:43387 (local address 127.0.0.1:60126)
0810 18:09:01.261813 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.261863 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 18:09:01.262116 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.125:43387: Network error: Client connection negotiation failed: client connection to 127.21.182.125:43387: connect: Connection refused (error 111)
I20260810 18:09:01.330293 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } attempt: 2
I20260810 18:09:01.331254 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.330879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:43387 (local address 127.0.0.1:60128)
0810 18:09:01.331065 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.331144 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 18:09:01.332139 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.331838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:43387 (local address 127.0.0.1:60130)
0810 18:09:01.332011 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.332059 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:43387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260810 18:09:01.332345 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.125:43387: Network error: Client connection negotiation failed: client connection to 127.21.182.125:43387: connect: Connection refused (error 111)
W20260810 18:09:01.336958 4868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:01.337256 4868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:01.337340 4868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:01.341621 4868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:09:01.341681 4868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:01.341701 4868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:01.341753 4868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:09:01.341795 4868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:09:01.347365 4868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:41415,127.21.182.125:43387,127.21.182.124:33679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.125.keytab
--principal=kudu/127.21.182.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.125:43387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.182.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:01.348577 4868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:01.349707 4868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:01.358708 4878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:01.358743 4874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:01.358743 4875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:01.359081 4868 server_base.cc:1034] running on GCE node
I20260810 18:09:01.359965 4868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:01.361225 4868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:01.362406 4868 hybrid_clock.cc:648] HybridClock initialized: now 1786385341362391 us; error 33 us; skew 500 ppm
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385341, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:01.365581 4868 init.cc:377] Logged in from keytab as kudu/127.21.182.125@KRBTEST.COM (short username kudu)
I20260810 18:09:01.366654 4868 webserver.cc:460] Webserver started at http://127.21.182.125:40109/ using document root <none> and password file <none>
I20260810 18:09:01.367292 4868 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:01.367367 4868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:01.367635 4868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:09:01.369979 4868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/data/instance:
uuid: "5be6fd4c3480417fb3d1e42645eabd56"
format_stamp: "Formatted at 2026-08-10 18:09:01 on dist-test-slave-27sr"
server_key: "21982afbdd3585fd7a8123cf49c17466"
server_key_iv: "c7b712de5bbadb65198453aa4891b9f9"
server_key_version: "encryptionkey@0"
I20260810 18:09:01.370627 4868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/wal/instance:
uuid: "5be6fd4c3480417fb3d1e42645eabd56"
format_stamp: "Formatted at 2026-08-10 18:09:01 on dist-test-slave-27sr"
server_key: "21982afbdd3585fd7a8123cf49c17466"
server_key_iv: "c7b712de5bbadb65198453aa4891b9f9"
server_key_version: "encryptionkey@0"
I20260810 18:09:01.375106 4868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 18:09:01.378134 4887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:01.380070 4868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:01.380285 4868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/wal
uuid: "5be6fd4c3480417fb3d1e42645eabd56"
format_stamp: "Formatted at 2026-08-10 18:09:01 on dist-test-slave-27sr"
server_key: "21982afbdd3585fd7a8123cf49c17466"
server_key_iv: "c7b712de5bbadb65198453aa4891b9f9"
server_key_version: "encryptionkey@0"
I20260810 18:09:01.380450 4868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:01.394548 4868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:01.399309 4868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:01.399552 4868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:01.407366 4868 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.125:43387
I20260810 18:09:01.407703 4946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.125:43387 every 8 connection(s)
I20260810 18:09:01.408710 4868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/data/info.pb
I20260810 18:09:01.411866 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:01.414194 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 4868
I20260810 18:09:01.414359 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-1/wal/instance
I20260810 18:09:01.414695 22233 external_mini_cluster.cc:1428] Setting key 0bb200d1f71fafd750ab09e563eb5e4c
I20260810 18:09:01.417255 4947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:09:01.423547 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } attempt: 3
WARNING: no policy specified for kudu/127.21.182.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.124@KRBTEST.COM" created.
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385341, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.125@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385341, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for kudu/127.21.182.125@KRBTEST.COM
I20260810 18:09:01.435966 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.419378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.21.182.126:41415)
0810 18:09:01.419551 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.419557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.420619 (+ 1062us) server_negotiation.cc:407] Connection header received
0810 18:09:01.421654 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.421661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.421745 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.421862 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.423274 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.424220 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.425993 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.426406 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.426511 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.429969 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.429972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.429974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.430000 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.432824 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.433593 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.433599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.433602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.433713 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.434253 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.434259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.434263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.434558 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.434672 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.435479 (+ 807us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.435652 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:09:01.436007 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.424131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:43387 (local address 127.0.0.1:36736)
0810 18:09:01.424369 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.424388 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:01.424475 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:01.425160 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:01.425164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:01.425392 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:01.425520 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.425528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.427048 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.427053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.427970 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.427984 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.428990 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.428997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.429179 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.429770 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:01.429793 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:01.431556 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:01.434032 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.434036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.434040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.434339 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.434750 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.434755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.434757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.434853 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.435430 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:01.435433 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:01.435542 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 18:09:01.435634 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 18:09:01.435792 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260810 18:09:01.437075 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.424243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.21.182.125:43387)
0810 18:09:01.424731 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.424740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.424762 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:09:01.424799 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.424803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.424976 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.425114 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.425678 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.426916 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.428123 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.428802 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.429470 (+ 668us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.431754 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.431761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.431776 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.431802 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.433918 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.434473 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.434482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.434484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.434580 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.434970 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.434973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.434975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.435286 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.436155 (+ 869us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.436280 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.436441 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":398,"spinlock_wait_cycles":2816,"thread_start_us":144,"threads_started":1}
I20260810 18:09:01.437075 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.419829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:41415 (local address 127.0.0.1:56560)
0810 18:09:01.420429 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.420485 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:01.421418 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:01.422035 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:01.422042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:01.422441 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:01.423079 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.423095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.424424 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.424430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.425841 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.425852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.426570 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.426573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.426737 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.428064 (+ 1327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:01.428092 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:01.429801 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:01.432982 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.432996 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.433012 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.433456 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.433846 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.433852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.433858 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.434107 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.434707 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:01.434719 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:01.435116 (+ 397us) client_negotiation.cc:769] Sending connection context
0810 18:09:01.435390 (+ 274us) client_negotiation.cc:240] Negotiation successful
0810 18:09:01.435803 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":55,"spinlock_wait_cycles":8576,"thread_start_us":154,"threads_started":1}
I20260810 18:09:01.439536 4947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:09:01.439637 4860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:09:01.440536 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.440230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52394)
0810 18:09:01.440420 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.440469 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260810 18:09:01.441430 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.441050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52410)
0810 18:09:01.441258 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.441346 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260810 18:09:01.441593 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:33679: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385341, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.125@KRBTEST.COM for kudu/127.21.182.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.124.keytab.
Entry for principal kudu/127.21.182.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.124.keytab.
I20260810 18:09:01.447665 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.124:33679
--webserver_interface=127.21.182.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.124.keytab
--principal=kudu/127.21.182.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:41415,127.21.182.125:43387,127.21.182.124:33679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:09:01.449828 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.440230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:43387 (local address 127.0.0.1:36746)
0810 18:09:01.440429 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.440445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:01.440548 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:01.440840 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:01.440843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:01.441058 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:01.441217 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.441230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.442443 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.442478 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.443078 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.443088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.443554 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.443556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.443681 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.444359 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:01.444388 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:01.445904 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:01.448131 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.448136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.448139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.448366 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.448693 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.448697 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.448699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.448792 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.449217 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:01.449220 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:01.449324 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 18:09:01.449439 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 18:09:01.449615 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260810 18:09:01.449828 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.440307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.21.182.125:43387)
0810 18:09:01.440431 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.440438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.440494 (+ 56us) server_negotiation.cc:407] Connection header received
0810 18:09:01.440655 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.440661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.440707 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.440814 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.441417 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.442307 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.443194 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.443422 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.443596 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.446087 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.446091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.446093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.446116 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.448009 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.448478 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.448483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.448485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.448570 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.448892 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.448898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.448900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.449105 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.449227 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.449505 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.449688 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":45}
I20260810 18:09:01.450819 4947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:09:01.451650 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.451348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52418)
0810 18:09:01.451486 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.451580 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 18:09:01.452045 4888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
I20260810 18:09:01.453112 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.452882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52428)
0810 18:09:01.452999 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.453049 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 18:09:01.453295 4947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:33679: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
I20260810 18:09:01.472790 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } attempt: 1
I20260810 18:09:01.473928 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.473540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52440)
0810 18:09:01.473726 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.473813 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 18:09:01.474815 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.474546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52456)
0810 18:09:01.474675 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.474737 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 18:09:01.474969 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:33679: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
I20260810 18:09:01.502599 4947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } attempt: 1
I20260810 18:09:01.503749 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.503355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52472)
0810 18:09:01.503556 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.503643 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 18:09:01.504761 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.504482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52484)
0810 18:09:01.504607 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.504675 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 18:09:01.504988 4947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:33679: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
I20260810 18:09:01.550150 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } attempt: 2
I20260810 18:09:01.551123 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.550807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52492)
0810 18:09:01.550976 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.551042 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 18:09:01.552049 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.551670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52496)
0810 18:09:01.551851 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.551967 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260810 18:09:01.552326 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:33679: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
I20260810 18:09:01.573201 4947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } attempt: 2
W20260810 18:09:01.573606 4954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:01.573933 4954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:01.574046 4954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 18:09:01.574251 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.573824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52508)
0810 18:09:01.574065 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.574144 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260810 18:09:01.575174 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.574902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52516)
0810 18:09:01.575026 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.575104 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 18:09:01.575338 4947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:33679: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33679: connect: Connection refused (error 111)
W20260810 18:09:01.578778 4954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:09:01.578841 4954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:01.578861 4954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:01.578879 4954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:09:01.578895 4954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:09:01.585583 4954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:41415,127.21.182.125:43387,127.21.182.124:33679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.124.keytab
--principal=kudu/127.21.182.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.124:33679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.182.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:01.586997 4954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:01.588305 4954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:01.598194 4962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:01.598194 4964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:01.598212 4961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:01.598582 4954 server_base.cc:1034] running on GCE node
I20260810 18:09:01.599311 4954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:01.600572 4954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:01.601756 4954 hybrid_clock.cc:648] HybridClock initialized: now 1786385341601734 us; error 47 us; skew 500 ppm
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385341, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:01.604867 4954 init.cc:377] Logged in from keytab as kudu/127.21.182.124@KRBTEST.COM (short username kudu)
I20260810 18:09:01.605826 4954 webserver.cc:460] Webserver started at http://127.21.182.124:37855/ using document root <none> and password file <none>
I20260810 18:09:01.606385 4954 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:01.606446 4954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:01.606626 4954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:09:01.608836 4954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/data/instance:
uuid: "eda2e4a8c37743208461b956e82dd281"
format_stamp: "Formatted at 2026-08-10 18:09:01 on dist-test-slave-27sr"
server_key: "cc737f783e7f44b9d69faf96f03057d7"
server_key_iv: "4d42649be1dc4f87ab7eb4e1d32781da"
server_key_version: "encryptionkey@0"
I20260810 18:09:01.609390 4954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/wal/instance:
uuid: "eda2e4a8c37743208461b956e82dd281"
format_stamp: "Formatted at 2026-08-10 18:09:01 on dist-test-slave-27sr"
server_key: "cc737f783e7f44b9d69faf96f03057d7"
server_key_iv: "4d42649be1dc4f87ab7eb4e1d32781da"
server_key_version: "encryptionkey@0"
I20260810 18:09:01.613344 4954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:09:01.616142 4971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:01.617516 4954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:01.617655 4954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/wal
uuid: "eda2e4a8c37743208461b956e82dd281"
format_stamp: "Formatted at 2026-08-10 18:09:01 on dist-test-slave-27sr"
server_key: "cc737f783e7f44b9d69faf96f03057d7"
server_key_iv: "4d42649be1dc4f87ab7eb4e1d32781da"
server_key_version: "encryptionkey@0"
I20260810 18:09:01.617784 4954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:01.639479 4954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:01.643190 4954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:01.643417 4954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:01.651505 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } attempt: 3
I20260810 18:09:01.651453 4954 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.124:33679
I20260810 18:09:01.651485 5034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.124:33679 every 8 connection(s)
I20260810 18:09:01.653183 4954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/data/info.pb
I20260810 18:09:01.656258 5036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:01.656692 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 4954
I20260810 18:09:01.656826 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786385164390354-22233-0/minicluster-data/master-2/wal/instance
I20260810 18:09:01.657143 22233 external_mini_cluster.cc:1428] Setting key e659555214556e93fcb585bcda1a7dfd
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385341, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for kudu/127.21.182.124@KRBTEST.COM
I20260810 18:09:01.661412 5036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.124@KRBTEST.COM
I20260810 18:09:01.666764 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.652232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52520)
0810 18:09:01.652469 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.652485 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:01.652586 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:01.655159 (+ 2573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:01.655163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:01.655390 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:01.655512 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.655520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.657256 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.657263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.657970 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.657982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.659162 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.659169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.659284 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.659916 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:01.659934 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:01.661566 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:01.664272 (+ 2706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.664278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.664280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.664542 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.664993 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.664997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.664999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.665096 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.665885 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:01.665892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:01.666020 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 18:09:01.666391 (+ 371us) client_negotiation.cc:240] Negotiation successful
0810 18:09:01.666563 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260810 18:09:01.666956 4947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } attempt: 3
I20260810 18:09:01.667768 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.653224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.21.182.124:33679)
0810 18:09:01.653769 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.653791 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.653839 (+ 48us) server_negotiation.cc:407] Connection header received
0810 18:09:01.654501 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.654517 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.654868 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.655223 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.655683 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.657084 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.658260 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.659004 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.659183 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.661775 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.661781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.661800 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.661832 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.664101 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.664699 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.664705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.664711 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.664865 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.665232 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.665235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.665236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.665735 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.665901 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.666331 (+ 430us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.666677 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":157,"threads_started":1}
I20260810 18:09:01.669080 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.658161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52530 (local address 127.21.182.124:33679)
0810 18:09:01.658534 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.658540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.658551 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:09:01.658579 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.658581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.658622 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.658742 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.659650 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.660519 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.661358 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.661592 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.661704 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.665231 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.665235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.665237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.665295 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.667077 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.667583 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.667587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.667589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.667675 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.668037 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.668040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.668042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.668248 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.668412 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.668735 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.668849 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"spinlock_wait_cycles":896,"thread_start_us":143,"threads_started":1}
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385341, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.124@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:01.673596 4860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c: Bootstrap starting.
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385341, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.125@KRBTEST.COM for kudu/127.21.182.124@KRBTEST.COM
I20260810 18:09:01.676941 4860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:01.677977 4860 log.cc:824] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:01.678237 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.663613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.21.182.126:41415)
0810 18:09:01.663772 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.663777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.664312 (+ 535us) server_negotiation.cc:407] Connection header received
0810 18:09:01.664515 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.664519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.664565 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.664673 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.665440 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.666146 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.667636 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.667980 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.668060 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.672339 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.672343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.672346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.672374 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.674306 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.676030 (+ 1724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.676035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.676037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.676138 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.676788 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.676794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.676799 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.677351 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.677444 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.677947 (+ 503us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.678087 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:09:01.678930 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.663574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:41415 (local address 127.0.0.1:56566)
0810 18:09:01.664185 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.664237 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:01.664410 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:01.664812 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:01.664817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:01.665133 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:01.665301 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.665308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.666248 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.666254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.667500 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.667513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.668337 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.668340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.668439 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.668974 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:01.668991 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:01.672192 (+ 3201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:01.674520 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.674536 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.675556 (+ 1020us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.675937 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.676263 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.676266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.676268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.676644 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.677520 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:01.677526 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:01.677660 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 18:09:01.678442 (+ 782us) client_negotiation.cc:240] Negotiation successful
0810 18:09:01.678695 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":128,"threads_started":1}
I20260810 18:09:01.680495 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.667532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52536)
0810 18:09:01.667812 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.667827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:01.667934 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:01.669667 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:01.669670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:01.669877 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:01.670014 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.670026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.671160 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.671164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.672050 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.672067 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.673089 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.673096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.673213 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.673978 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:01.673999 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:01.675436 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:01.678068 (+ 2632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.678072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.678075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.678373 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.679280 (+ 907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.679286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.679291 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.679410 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.679913 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:01.679917 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:01.680034 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 18:09:01.680150 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 18:09:01.680324 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260810 18:09:01.680495 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.668092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52536 (local address 127.21.182.124:33679)
0810 18:09:01.669281 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.669286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.669300 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:09:01.669453 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.669456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.669503 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.669588 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.670187 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.671008 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.672213 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.672595 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.674604 (+ 2009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.676083 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.676089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.676095 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.676121 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.677926 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.678801 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.678805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.678810 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.678889 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.679553 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.679557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.679560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.679792 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.679927 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.680227 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.680347 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":811,"server-negotiator.queue_time_us":1129}
I20260810 18:09:01.680596 4860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c: No bootstrap required, opened a new log
I20260810 18:09:01.681327 5036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:09:01.683509 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.671206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.21.182.126:41415)
0810 18:09:01.671877 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.671913 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.671927 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:09:01.671967 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.671982 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.672037 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.672916 (+ 879us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.673013 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.673779 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.674850 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.675250 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.675355 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.678723 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.678728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.678734 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.678766 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.681541 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.682060 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.682064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.682066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.682170 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.682479 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.682482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.682484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.682747 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.682957 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.683171 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.683296 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":72,"threads_started":1}
I20260810 18:09:01.685309 4860 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } }
I20260810 18:09:01.685557 4860 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:01.685616 4860 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e83f42a0f44219be9ab6f22b02c98c, State: Initialized, Role: FOLLOWER
I20260810 18:09:01.686139 4860 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } }
I20260810 18:09:01.686765 5044 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } } }
I20260810 18:09:01.686936 5044 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:09:01.687074 4947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56: Bootstrap starting.
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385341, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.124@KRBTEST.COM for kudu/127.21.182.125@KRBTEST.COM
I20260810 18:09:01.687652 4860 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:09:01.690425 4947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:01.691438 4947 log.cc:824] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:01.692263 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.682075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36762 (local address 127.21.182.125:43387)
0810 18:09:01.682229 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.682236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.682290 (+ 54us) server_negotiation.cc:407] Connection header received
0810 18:09:01.682425 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.682431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.682489 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.682559 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.683094 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.683862 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.685103 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.685445 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.685515 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.688260 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.688264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.688266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.688292 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.690229 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.690794 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.690798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.690800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.690895 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.691215 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.691218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.691220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.691470 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.691542 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.691956 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.692078 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:09:01.693075 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.681836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:43387 (local address 127.0.0.1:36762)
0810 18:09:01.682227 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.682244 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:01.682332 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:01.682581 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:01.682584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:01.682767 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:01.682910 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.682917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.683995 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.683998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.684949 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.684960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.685730 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.685734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.685823 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.686444 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:01.686455 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:01.688066 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:01.690382 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.690385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.690388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.690672 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.691012 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.691015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.691017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.691103 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.691622 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:01.691626 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:01.691739 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 18:09:01.692450 (+ 711us) client_negotiation.cc:240] Negotiation successful
0810 18:09:01.692913 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321}
I20260810 18:09:01.693739 5036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.125@KRBTEST.COM
W20260810 18:09:01.696028 5058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:09:01.696103 5058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:09:01.693706 4787 master_runner.cc:186] Error getting master registration for 127.21.182.125:43387: OK, instance_id { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" instance_seqno: 1786385341405158 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:09:01.698408 4787 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:09:01.699569 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.684922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.21.182.125:43387)
0810 18:09:01.688209 (+ 3287us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.688224 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.688237 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:09:01.688265 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.688268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.688311 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.689376 (+ 1065us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.689548 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.690382 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.691646 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.692059 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.692164 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.695135 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.695139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.695141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.695175 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.697106 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.697691 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.697695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.697696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.697792 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.698125 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.698129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.698131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.698381 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.698586 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.699254 (+ 668us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.699384 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3195,"thread_start_us":130,"threads_started":1}
I20260810 18:09:01.702139 4947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56: No bootstrap required, opened a new log
I20260810 18:09:01.703217 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Aug 10 18:09:01 dist-test-slave-27sr krb5kdc[4772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385341, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.124@KRBTEST.COM for kudu/127.21.182.124@KRBTEST.COM
I20260810 18:09:01.707603 4947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } }
I20260810 18:09:01.707945 4947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:01.708020 4947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5be6fd4c3480417fb3d1e42645eabd56, State: Initialized, Role: FOLLOWER
I20260810 18:09:01.708478 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.698681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.21.182.124:33679)
0810 18:09:01.698821 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.698826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.698839 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:09:01.698878 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.698881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.698926 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.699009 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.699446 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.700249 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.701642 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.701935 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.702015 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.704681 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.704685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.704689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.704719 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.706477 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.706972 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.706976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.706977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.707058 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.707362 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.707368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.707373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.707717 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.707803 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.708177 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.708303 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:09:01.708608 4947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } }
I20260810 18:09:01.708997 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.694197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33679 (local address 127.0.0.1:52552)
0810 18:09:01.694521 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:01.694536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:01.694623 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:01.699121 (+ 4498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:01.699123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:01.699270 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:01.699330 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.699334 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.700608 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.700612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.701294 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:01.701305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.702171 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.702175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:01.702336 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.702881 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:01.702895 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:01.704512 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:01.706598 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.706601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.706604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.706870 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.707160 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:01.707163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:01.707165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:01.707261 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:01.707864 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:01.707868 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:01.708003 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 18:09:01.708675 (+ 672us) client_negotiation.cc:240] Negotiation successful
0810 18:09:01.708827 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":92,"threads_started":1}
I20260810 18:09:01.710176 4947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:09:01.712407 5036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281: Bootstrap starting.
I20260810 18:09:01.716867 5036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:01.717880 5036 log.cc:824] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:01.720170 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.706038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36790 (local address 127.21.182.125:43387)
0810 18:09:01.706167 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.706172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.708366 (+ 2194us) server_negotiation.cc:407] Connection header received
0810 18:09:01.708554 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.708559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.708615 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.708669 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.710219 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.710971 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.712299 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.712655 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.712747 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.716157 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.716163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.716167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.716201 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.718044 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.718571 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.718576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.718578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.718709 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.719069 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.719074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.719075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.719334 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.719469 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.719842 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.720010 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:09:01.719523 4868 master_runner.cc:186] Error getting master registration for 127.21.182.124:33679: OK, instance_id { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" instance_seqno: 1786385341648968 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:09:01.722802 4868 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:09:01.724264 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.706870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.21.182.126:41415)
0810 18:09:01.706996 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.707000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.712248 (+ 5248us) server_negotiation.cc:407] Connection header received
0810 18:09:01.712478 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.712482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.712541 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.712598 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.716207 (+ 3609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.717057 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.718301 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.718679 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.718766 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.720545 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.720549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.720552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.720579 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.722312 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.722848 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.722852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.722854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.722988 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.723310 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.723313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.723315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.723531 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.723682 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.723953 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.724084 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:09:01.725060 5036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281: No bootstrap required, opened a new log
I20260810 18:09:01.728240 5072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } } }
I20260810 18:09:01.728399 5072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:09:01.728338 5036 raft_consensus.cc:348] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } }
I20260810 18:09:01.728614 5036 raft_consensus.cc:374] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:01.728690 5036 raft_consensus.cc:729] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eda2e4a8c37743208461b956e82dd281, State: Initialized, Role: FOLLOWER
I20260810 18:09:01.729269 5036 consensus_queue.cc:260] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } }
I20260810 18:09:01.730702 5036 sys_catalog.cc:564] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 18:09:01.732228 5086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:09:01.732311 5086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:09:01.733819 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:01.707550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.21.182.124:33679)
0810 18:09:01.707648 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:01.707652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:01.713682 (+ 6030us) server_negotiation.cc:407] Connection header received
0810 18:09:01.713874 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:01.713878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:01.713927 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:01.713979 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:01.719511 (+ 5532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.720402 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.721585 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:01.721977 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:01.722065 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:01.729566 (+ 7501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:01.729571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:01.729574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:01.729602 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:01.731284 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.731819 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.731823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.731825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.731948 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:01.732252 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:01.732255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:01.732257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:01.732466 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:01.733079 (+ 613us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:01.733548 (+ 469us) server_negotiation.cc:299] Negotiation successful
0810 18:09:01.733673 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 18:09:01.732853 5089 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } } }
I20260810 18:09:01.735807 5089 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 18:09:01.736099 5102 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:09:01.736181 5102 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:09:01.739115 4954 master_runner.cc:263] No leader master found from master eda2e4a8c37743208461b956e82dd281
I20260810 18:09:01.739192 4954 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 18:09:01.894991 4979 tablet.cc:2369] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:09:01.901414 4893 tablet.cc:2369] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:09:01.919677 4809 tablet.cc:2369] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:09:01.974289 5072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:09:01.974516 5072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } }
I20260810 18:09:01.975507 5072 leader_election.cc:288] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8e83f42a0f44219be9ab6f22b02c98c (127.21.182.126:41415), eda2e4a8c37743208461b956e82dd281 (127.21.182.124:33679)
I20260810 18:09:01.976327 5004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5be6fd4c3480417fb3d1e42645eabd56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eda2e4a8c37743208461b956e82dd281" is_pre_election: true
I20260810 18:09:01.976330 4831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5be6fd4c3480417fb3d1e42645eabd56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" is_pre_election: true
I20260810 18:09:01.976670 5004 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5be6fd4c3480417fb3d1e42645eabd56 in term 0.
I20260810 18:09:01.976692 4831 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5be6fd4c3480417fb3d1e42645eabd56 in term 0.
I20260810 18:09:01.977135 4888 leader_election.cc:302] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5be6fd4c3480417fb3d1e42645eabd56, c8e83f42a0f44219be9ab6f22b02c98c; no voters:
I20260810 18:09:01.977465 5072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:09:01.977571 5072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:09:01.977654 5072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:01.978822 5072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } }
I20260810 18:09:01.979188 5072 leader_election.cc:288] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [CANDIDATE]: Term 1 election: Requested vote from peers c8e83f42a0f44219be9ab6f22b02c98c (127.21.182.126:41415), eda2e4a8c37743208461b956e82dd281 (127.21.182.124:33679)
I20260810 18:09:01.979678 5004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5be6fd4c3480417fb3d1e42645eabd56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eda2e4a8c37743208461b956e82dd281"
I20260810 18:09:01.979729 4831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5be6fd4c3480417fb3d1e42645eabd56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8e83f42a0f44219be9ab6f22b02c98c"
I20260810 18:09:01.979825 5004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:01.979923 4831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:01.981009 5004 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5be6fd4c3480417fb3d1e42645eabd56 in term 1.
I20260810 18:09:01.981092 4831 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5be6fd4c3480417fb3d1e42645eabd56 in term 1.
I20260810 18:09:01.981379 4888 leader_election.cc:302] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5be6fd4c3480417fb3d1e42645eabd56, eda2e4a8c37743208461b956e82dd281; no voters:
I20260810 18:09:01.981590 5072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:09:01.981866 5072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [term 1 LEADER]: Becoming Leader. State: Replica: 5be6fd4c3480417fb3d1e42645eabd56, State: Running, Role: LEADER
I20260810 18:09:01.982210 5072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } }
I20260810 18:09:01.984405 5104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5be6fd4c3480417fb3d1e42645eabd56. Latest consensus state: current_term: 1 leader_uuid: "5be6fd4c3480417fb3d1e42645eabd56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } } }
I20260810 18:09:01.984536 5104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:01.984958 5107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:09:01.986107 5107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:09:01.992429 4831 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [term 1 FOLLOWER]: Refusing update from remote peer 5be6fd4c3480417fb3d1e42645eabd56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:09:01.992498 5004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [term 1 FOLLOWER]: Refusing update from remote peer 5be6fd4c3480417fb3d1e42645eabd56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:09:01.993214 5104 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:09:01.993402 5072 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:09:01.998016 5089 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5be6fd4c3480417fb3d1e42645eabd56. Latest consensus state: current_term: 1 leader_uuid: "5be6fd4c3480417fb3d1e42645eabd56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } } }
I20260810 18:09:01.998123 5089 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:09:01.998522 5044 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5be6fd4c3480417fb3d1e42645eabd56. Latest consensus state: current_term: 1 leader_uuid: "5be6fd4c3480417fb3d1e42645eabd56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } } }
I20260810 18:09:01.998631 5044 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:09:02.001008 5072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5be6fd4c3480417fb3d1e42645eabd56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } } }
I20260810 18:09:02.001132 5072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:02.001806 5072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5be6fd4c3480417fb3d1e42645eabd56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } } }
I20260810 18:09:02.001921 5072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56 [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:02.002249 5089 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5be6fd4c3480417fb3d1e42645eabd56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } } }
I20260810 18:09:02.002367 5089 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:09:02.002887 5107 catalog_manager.cc:1347] Generated new cluster ID: 31a4c512adfb4298aa3af6c945397773
I20260810 18:09:02.002964 5107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:09:02.002952 5044 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5be6fd4c3480417fb3d1e42645eabd56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e83f42a0f44219be9ab6f22b02c98c" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 41415 } } peers { permanent_uuid: "5be6fd4c3480417fb3d1e42645eabd56" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 43387 } } peers { permanent_uuid: "eda2e4a8c37743208461b956e82dd281" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33679 } } }
I20260810 18:09:02.003064 5044 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:09:02.017098 5107 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:09:02.018472 5107 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:09:02.025765 5107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5be6fd4c3480417fb3d1e42645eabd56: Generated new TSK 0
I20260810 18:09:02.026501 5107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:09:02.697513 5058 catalog_manager.cc:1259] Loaded cluster ID: 31a4c512adfb4298aa3af6c945397773
I20260810 18:09:02.697604 5058 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c: loading cluster ID for follower catalog manager: success
I20260810 18:09:02.699687 5058 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c: acquiring CA information for follower catalog manager: success
I20260810 18:09:02.700965 5058 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8e83f42a0f44219be9ab6f22b02c98c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:09:02.737911 5102 catalog_manager.cc:1259] Loaded cluster ID: 31a4c512adfb4298aa3af6c945397773
I20260810 18:09:02.738047 5102 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281: loading cluster ID for follower catalog manager: success
I20260810 18:09:02.739825 5102 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281: acquiring CA information for follower catalog manager: success
I20260810 18:09:02.741097 5102 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eda2e4a8c37743208461b956e82dd281: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:09:03.102065 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 4787
I20260810 18:09:03.113415 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 4868
I20260810 18:09:03.124056 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 4954
2026-08-10T18:09:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3455 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:09:03 dist-test-slave-27sr krb5kdc[5123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:09:03 dist-test-slave-27sr krb5kdc[5123](info): set up 2 sockets
Aug 10 18:09:03 dist-test-slave-27sr krb5kdc[5123](info): commencing operation
krb5kdc: starting...
W20260810 18:09:04.264853 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:09:04 dist-test-slave-27sr krb5kdc[5123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:09:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:09:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:09:04.407564 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:33557
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:37781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:09:04.533946 5138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:04.534315 5138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:04.534400 5138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:04.538753 5138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:09:04.538872 5138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:04.538908 5138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:04.538949 5138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:09:04.539008 5138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:09:04.544790 5138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:37781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:33557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:04.546056 5138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:04.547256 5138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:04.556644 5146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:04.556749 5143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:04.556783 5144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:04.556989 5138 server_base.cc:1034] running on GCE node
I20260810 18:09:04.557772 5138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:04.559243 5138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:04.560483 5138 hybrid_clock.cc:648] HybridClock initialized: now 1786385344560459 us; error 35 us; skew 500 ppm
Aug 10 18:09:04 dist-test-slave-27sr krb5kdc[5123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385344, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:04.564105 5138 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:09:04.565297 5138 webserver.cc:460] Webserver started at http://127.21.182.126:34725/ using document root <none> and password file <none>
I20260810 18:09:04.566013 5138 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:04.566129 5138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:04.566412 5138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:09:04.569027 5138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "32377950dafe4bd3b4ae2a8f4599b627"
format_stamp: "Formatted at 2026-08-10 18:09:04 on dist-test-slave-27sr"
server_key: "ee14128a7b73c0f5264f96a937d08404"
server_key_iv: "c59b9e49bd09f5156c4594ea7c28bd68"
server_key_version: "encryptionkey@0"
I20260810 18:09:04.569842 5138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "32377950dafe4bd3b4ae2a8f4599b627"
format_stamp: "Formatted at 2026-08-10 18:09:04 on dist-test-slave-27sr"
server_key: "ee14128a7b73c0f5264f96a937d08404"
server_key_iv: "c59b9e49bd09f5156c4594ea7c28bd68"
server_key_version: "encryptionkey@0"
I20260810 18:09:04.574637 5138 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:09:04.577864 5154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:04.579530 5138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:09:04.579754 5138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "32377950dafe4bd3b4ae2a8f4599b627"
format_stamp: "Formatted at 2026-08-10 18:09:04 on dist-test-slave-27sr"
server_key: "ee14128a7b73c0f5264f96a937d08404"
server_key_iv: "c59b9e49bd09f5156c4594ea7c28bd68"
server_key_version: "encryptionkey@0"
I20260810 18:09:04.579981 5138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:04.612000 5138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:04.616235 5138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:04.616487 5138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:04.625162 5138 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:33557
I20260810 18:09:04.625169 5216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:33557 every 8 connection(s)
I20260810 18:09:04.626325 5138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:09:04.627848 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 5138
I20260810 18:09:04.628058 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:09:04.628433 22233 external_mini_cluster.cc:1428] Setting key c43e38a05159eadf0c65bc831dfaae2e
I20260810 18:09:04.629935 5217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:04.636967 5217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627: Bootstrap starting.
Aug 10 18:09:04 dist-test-slave-27sr krb5kdc[5123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:04.639755 5217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:04.640583 5217 log.cc:824] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:04.643147 5217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627: No bootstrap required, opened a new log
I20260810 18:09:04.645357 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:04.630090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.21.182.126:33557)
0810 18:09:04.630562 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:04.630581 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:04.630623 (+ 42us) server_negotiation.cc:407] Connection header received
0810 18:09:04.631335 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:04.631353 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:04.631715 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:04.632106 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:04.633075 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:04.634207 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:04.635418 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:04.636059 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:04.636200 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:04.639351 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:04.639357 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:04.639373 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:04.639407 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:04.641651 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:04.642156 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:04.642163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:04.642169 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:04.642323 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:04.642620 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:04.642623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:04.642625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:04.643176 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:04.643365 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:04.643696 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 18:09:04.644084 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":156,"threads_started":1}
I20260810 18:09:04.647250 5217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32377950dafe4bd3b4ae2a8f4599b627" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33557 } }
I20260810 18:09:04.647575 5217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:04.647663 5217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32377950dafe4bd3b4ae2a8f4599b627, State: Initialized, Role: FOLLOWER
I20260810 18:09:04.648185 5217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32377950dafe4bd3b4ae2a8f4599b627" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33557 } }
I20260810 18:09:04.648357 5217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:09:04.648425 5217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:09:04.648527 5217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:04.648720 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:09:04.649540 5217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32377950dafe4bd3b4ae2a8f4599b627" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33557 } }
I20260810 18:09:04.649957 5217 leader_election.cc:302] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32377950dafe4bd3b4ae2a8f4599b627; no voters:
I20260810 18:09:04.650349 5217 leader_election.cc:288] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:09:04.650497 5223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:09:04.650851 5223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [term 1 LEADER]: Becoming Leader. State: Replica: 32377950dafe4bd3b4ae2a8f4599b627, State: Running, Role: LEADER
I20260810 18:09:04.651207 5223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32377950dafe4bd3b4ae2a8f4599b627" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33557 } }
I20260810 18:09:04.651681 5217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:09:04.653213 5224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32377950dafe4bd3b4ae2a8f4599b627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32377950dafe4bd3b4ae2a8f4599b627" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33557 } } }
I20260810 18:09:04.653374 5224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:04.653179 5225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32377950dafe4bd3b4ae2a8f4599b627. Latest consensus state: current_term: 1 leader_uuid: "32377950dafe4bd3b4ae2a8f4599b627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32377950dafe4bd3b4ae2a8f4599b627" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33557 } } }
I20260810 18:09:04.653530 5225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627 [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:04.653987 5236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:09:04.657344 5236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:09:04.663240 5236 catalog_manager.cc:1347] Generated new cluster ID: f59ae635c25447438d7ed73238c0e876
I20260810 18:09:04.663312 5236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:09:04.701076 5236 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:09:04.702370 5236 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:09:04.710657 5236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32377950dafe4bd3b4ae2a8f4599b627: Generated new TSK 0
I20260810 18:09:04.711511 5236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:09:04.756450 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 5138
2026-08-10T18:09:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1631 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:09:04 dist-test-slave-27sr krb5kdc[5246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:09:04 dist-test-slave-27sr krb5kdc[5246](info): set up 2 sockets
Aug 10 18:09:04 dist-test-slave-27sr krb5kdc[5246](info): commencing operation
krb5kdc: starting...
W20260810 18:09:05.907621 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:09:06 dist-test-slave-27sr krb5kdc[5246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:09:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:09:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:09:06.051620 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:44475
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:44475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:09:06.173812 5260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:06.174170 5260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:06.174253 5260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:06.178544 5260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:09:06.178680 5260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:06.178721 5260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:06.178787 5260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:09:06.178844 5260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:09:06.184831 5260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:44475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:44475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:06.186111 5260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:06.187428 5260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:06.197404 5265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:06.197412 5260 server_base.cc:1034] running on GCE node
W20260810 18:09:06.197556 5268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:06.197412 5266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:06.198319 5260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:06.199699 5260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:06.200929 5260 hybrid_clock.cc:648] HybridClock initialized: now 1786385346200918 us; error 36 us; skew 500 ppm
Aug 10 18:09:06 dist-test-slave-27sr krb5kdc[5246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385346, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:06.208122 5260 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:09:06.209398 5260 webserver.cc:460] Webserver started at http://127.21.182.126:38949/ using document root <none> and password file <none>
I20260810 18:09:06.210080 5260 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:06.210196 5260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:06.210467 5260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:09:06.212983 5260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "3e8fd4fcef724555a349913e33c10286"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "0d7047b7bd5f6f686bb56aeb696da719"
server_key_iv: "9241dc5ce1bd13fd817d0042be058a0c"
server_key_version: "encryptionkey@0"
I20260810 18:09:06.213666 5260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "3e8fd4fcef724555a349913e33c10286"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "0d7047b7bd5f6f686bb56aeb696da719"
server_key_iv: "9241dc5ce1bd13fd817d0042be058a0c"
server_key_version: "encryptionkey@0"
I20260810 18:09:06.217985 5260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:09:06.220934 5275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:06.222316 5260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:09:06.222479 5260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "3e8fd4fcef724555a349913e33c10286"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "0d7047b7bd5f6f686bb56aeb696da719"
server_key_iv: "9241dc5ce1bd13fd817d0042be058a0c"
server_key_version: "encryptionkey@0"
I20260810 18:09:06.222604 5260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:06.239154 5260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:06.242794 5260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:06.243018 5260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:06.250643 5260 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:44475
I20260810 18:09:06.250970 5336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:44475 every 8 connection(s)
I20260810 18:09:06.251735 5260 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:09:06.254665 5337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:06.260633 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 5260
I20260810 18:09:06.260708 5337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Bootstrap starting.
I20260810 18:09:06.260802 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:09:06.261170 22233 external_mini_cluster.cc:1428] Setting key 275a6d9d97754542419f40c143478d33
I20260810 18:09:06.263630 5337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:06.264479 5337 log.cc:824] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:06.266698 5337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: No bootstrap required, opened a new log
Aug 10 18:09:06 dist-test-slave-27sr krb5kdc[5246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:06.271443 5337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } }
I20260810 18:09:06.271713 5337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:06.271798 5337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8fd4fcef724555a349913e33c10286, State: Initialized, Role: FOLLOWER
I20260810 18:09:06.272368 5337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } }
I20260810 18:09:06.272606 5337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:09:06.272708 5337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:09:06.272827 5337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:06.274098 5337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } }
I20260810 18:09:06.274472 5337 leader_election.cc:302] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e8fd4fcef724555a349913e33c10286; no voters:
I20260810 18:09:06.274873 5337 leader_election.cc:288] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:09:06.274971 5342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:09:06.275151 5342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 1 LEADER]: Becoming Leader. State: Replica: 3e8fd4fcef724555a349913e33c10286, State: Running, Role: LEADER
I20260810 18:09:06.275475 5342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } }
I20260810 18:09:06.276348 5337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:09:06.277456 5343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e8fd4fcef724555a349913e33c10286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } } }
I20260810 18:09:06.277458 5344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e8fd4fcef724555a349913e33c10286. Latest consensus state: current_term: 1 leader_uuid: "3e8fd4fcef724555a349913e33c10286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } } }
I20260810 18:09:06.277684 5344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:06.277626 5343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:06.277956 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:06.262634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.21.182.126:44475)
0810 18:09:06.263116 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:06.263133 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:06.263171 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:09:06.263941 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:06.263962 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:06.264315 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:06.264633 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:06.265512 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:06.266542 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:06.267724 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:06.268297 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:06.268420 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:06.271439 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:06.271444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:06.271460 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:06.271489 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:06.273829 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:06.274351 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:06.274356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:06.274361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:06.274483 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:06.274818 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:06.274822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:06.274826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:06.275501 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:06.276390 (+ 889us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:06.276554 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 18:09:06.276884 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":155,"threads_started":1}
I20260810 18:09:06.278192 5352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:09:06.281827 5352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:09:06.288117 5352 catalog_manager.cc:1347] Generated new cluster ID: e063ee4ac5624bb48ccc37b3601d0a06
I20260810 18:09:06.288211 5352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:09:06.302773 5352 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:09:06.303812 5352 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:09:06.308915 5352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Generated new TSK 0
I20260810 18:09:06.309762 5352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:09:06.351473 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:44475
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:09:06.478448 5365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:06.478781 5365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:06.478829 5365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:06.483229 5365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:06.483320 5365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:06.483387 5365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:09:06.489364 5365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:44475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:06.490664 5365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:06.491856 5365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:06.502938 5374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:06.502935 5372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:06.503057 5365 server_base.cc:1034] running on GCE node
W20260810 18:09:06.503314 5371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:06.503799 5365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:06.504580 5365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:06.505779 5365 hybrid_clock.cc:648] HybridClock initialized: now 1786385346505751 us; error 43 us; skew 500 ppm
Aug 10 18:09:06 dist-test-slave-27sr krb5kdc[5246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385346, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:06.508975 5365 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:09:06.510179 5365 webserver.cc:460] Webserver started at http://127.21.182.65:38689/ using document root <none> and password file <none>
I20260810 18:09:06.510815 5365 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:06.510914 5365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:06.511178 5365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:09:06.513499 5365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "70c4da52e8d044108d1aff802619c4d5"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "fd1e782ae19e84525ea4f3e24c6b2c86"
server_key_iv: "9ae3a24352f62a47a50bc881cbed13b2"
server_key_version: "encryptionkey@0"
I20260810 18:09:06.514117 5365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "70c4da52e8d044108d1aff802619c4d5"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "fd1e782ae19e84525ea4f3e24c6b2c86"
server_key_iv: "9ae3a24352f62a47a50bc881cbed13b2"
server_key_version: "encryptionkey@0"
I20260810 18:09:06.518375 5365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:09:06.521116 5382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:06.522521 5365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:09:06.522686 5365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "70c4da52e8d044108d1aff802619c4d5"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "fd1e782ae19e84525ea4f3e24c6b2c86"
server_key_iv: "9ae3a24352f62a47a50bc881cbed13b2"
server_key_version: "encryptionkey@0"
I20260810 18:09:06.522820 5365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:06.550660 5365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:06.554064 5365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:06.554314 5365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:06.555013 5365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:06.556039 5365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:09:06.556110 5365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:06.556190 5365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:09:06.556231 5365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:06.566069 5365 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:45561
I20260810 18:09:06.566093 5523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:45561 every 8 connection(s)
I20260810 18:09:06.567166 5365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:09:06.570122 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 5365
I20260810 18:09:06.570298 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:09:06.570689 22233 external_mini_cluster.cc:1428] Setting key d7345200cbb4ae78748ed9c8664106ac
Aug 10 18:09:06 dist-test-slave-27sr krb5kdc[5246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385346, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
I20260810 18:09:06.584842 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:06.569350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:43425 (local address 127.21.182.126:44475)
0810 18:09:06.569520 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:06.569524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:06.570525 (+ 1001us) server_negotiation.cc:407] Connection header received
0810 18:09:06.572134 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:06.572140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:06.572194 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:06.572296 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:06.573657 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:06.574506 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:06.576495 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:06.576913 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:06.577051 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:06.580187 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:06.580194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:06.580201 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:06.580244 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:06.582221 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:06.582926 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:06.582934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:06.582939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:06.583051 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:06.583477 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:06.583481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:06.583483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:06.583772 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:06.583936 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:06.584571 (+ 635us) server_negotiation.cc:299] Negotiation successful
0810 18:09:06.584702 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 18:09:06.585809 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:06.569796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:43425)
0810 18:09:06.570327 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:06.570386 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:06.571827 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:06.572517 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:06.572525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:06.572956 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:06.573488 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:06.573502 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:06.574655 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:06.574657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:06.576345 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:06.576357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:06.577080 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:06.577083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:06.577426 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:06.578092 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:06.578110 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:06.579948 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:06.582378 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:06.582385 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:06.582403 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:06.582752 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:06.583162 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:06.583165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:06.583166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:06.583336 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:06.583985 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:06.583991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:06.584284 (+ 293us) client_negotiation.cc:769] Sending connection context
0810 18:09:06.584477 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 18:09:06.584829 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":157,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
I20260810 18:09:06.587455 5524 heartbeater.cc:347] Connected to a master server at 127.21.182.126:44475
I20260810 18:09:06.587803 5524 heartbeater.cc:464] Registering TS with master...
I20260810 18:09:06.588578 5524 heartbeater.cc:511] Master 127.21.182.126:44475 requested a full tablet report, sending...
I20260810 18:09:06.590952 5295 ts_manager.cc:194] Registered new tserver with Master: 70c4da52e8d044108d1aff802619c4d5 (127.21.182.65:45561)
I20260810 18:09:06.592458 5295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:43425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:09:06.608129 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:44475
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:09:06.726711 5529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:06.727046 5529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:06.727145 5529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:06.731292 5529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:06.731384 5529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:06.731472 5529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:09:06.736774 5529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:44475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:06.737905 5529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:06.738998 5529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:06.749722 5535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:06.749832 5534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:06.749864 5529 server_base.cc:1034] running on GCE node
W20260810 18:09:06.749724 5538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:06.750489 5529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:06.751227 5529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:06.752431 5529 hybrid_clock.cc:648] HybridClock initialized: now 1786385346752414 us; error 34 us; skew 500 ppm
Aug 10 18:09:06 dist-test-slave-27sr krb5kdc[5246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385346, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:06.755457 5529 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:09:06.756453 5529 webserver.cc:460] Webserver started at http://127.21.182.66:34321/ using document root <none> and password file <none>
I20260810 18:09:06.757056 5529 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:06.757121 5529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:06.757411 5529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:09:06.759534 5529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "26839255bc1c4be8ab44132c335dd72d"
server_key_iv: "ecb26a6e7cfeba55d6123c3c7d8c6742"
server_key_version: "encryptionkey@0"
I20260810 18:09:06.760165 5529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "26839255bc1c4be8ab44132c335dd72d"
server_key_iv: "ecb26a6e7cfeba55d6123c3c7d8c6742"
server_key_version: "encryptionkey@0"
I20260810 18:09:06.764158 5529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:09:06.766898 5546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:06.768368 5529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:06.768501 5529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "26839255bc1c4be8ab44132c335dd72d"
server_key_iv: "ecb26a6e7cfeba55d6123c3c7d8c6742"
server_key_version: "encryptionkey@0"
I20260810 18:09:06.768613 5529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:06.792094 5529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:06.795774 5529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:06.796025 5529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:06.796710 5529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:06.797657 5529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:09:06.797706 5529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:06.797780 5529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:09:06.797816 5529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:06.807003 5529 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:34303
I20260810 18:09:06.807032 5680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:34303 every 8 connection(s)
I20260810 18:09:06.808077 5529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:09:06.816522 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 5529
I20260810 18:09:06.816711 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:09:06.817116 22233 external_mini_cluster.cc:1428] Setting key 0ca9b87f963661c2816e39061977fd07
Aug 10 18:09:06 dist-test-slave-27sr krb5kdc[5246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385346, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:06.823695 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:06.810194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:38901 (local address 127.21.182.126:44475)
0810 18:09:06.810332 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:06.810335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:06.811318 (+ 983us) server_negotiation.cc:407] Connection header received
0810 18:09:06.812853 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:06.812856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:06.812931 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:06.813029 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:06.814246 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:06.814772 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:06.815948 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:06.816170 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:06.816305 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:06.819450 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:06.819456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:06.819462 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:06.819498 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:06.821310 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:06.821992 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:06.821994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:06.821996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:06.822076 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:06.822511 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:06.822514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:06.822515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:06.822693 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:06.822768 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:06.823461 (+ 693us) server_negotiation.cc:299] Negotiation successful
0810 18:09:06.823570 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:09:06.825111 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:06.810681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:38901)
0810 18:09:06.811161 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:06.811204 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:06.812603 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:06.813213 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:06.813220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:06.813609 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:06.814100 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:06.814113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:06.814913 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:06.814923 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:06.815799 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:06.815808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:06.816355 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:06.816360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:06.816525 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:06.817283 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:06.817313 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:06.819268 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:06.821442 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:06.821451 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:06.821472 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:06.821887 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:06.822190 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:06.822196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:06.822199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:06.822410 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:06.822803 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:06.822813 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:06.823189 (+ 376us) client_negotiation.cc:769] Sending connection context
0810 18:09:06.823424 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 18:09:06.823820 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260810 18:09:06.826578 5681 heartbeater.cc:347] Connected to a master server at 127.21.182.126:44475
I20260810 18:09:06.826892 5681 heartbeater.cc:464] Registering TS with master...
I20260810 18:09:06.827558 5681 heartbeater.cc:511] Master 127.21.182.126:44475 requested a full tablet report, sending...
I20260810 18:09:06.828953 5295 ts_manager.cc:194] Registered new tserver with Master: 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303)
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
I20260810 18:09:06.829668 5295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:38901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:09:06.852187 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:44475
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:09:06.977636 5686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:06.978022 5686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:06.978107 5686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:06.982501 5686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:06.982615 5686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:06.982720 5686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:09:06.988466 5686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:44475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:06.989756 5686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:06.990936 5686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:07.003547 5692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:07.003547 5691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:07.004155 5686 server_base.cc:1034] running on GCE node
W20260810 18:09:07.004352 5694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:07.004964 5686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:07.006089 5686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:07.007395 5686 hybrid_clock.cc:648] HybridClock initialized: now 1786385347007342 us; error 78 us; skew 500 ppm
Aug 10 18:09:07 dist-test-slave-27sr krb5kdc[5246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385347, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:07.011611 5686 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:09:07.012727 5686 webserver.cc:460] Webserver started at http://127.21.182.67:35299/ using document root <none> and password file <none>
I20260810 18:09:07.013388 5686 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:07.013489 5686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:07.013798 5686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:09:07.016181 5686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "e55952bf73b34a8ea5c0409283de7ab7"
format_stamp: "Formatted at 2026-08-10 18:09:07 on dist-test-slave-27sr"
server_key: "aac1d8e4bcb25fab835ccc0f4088c6c5"
server_key_iv: "b87129078146cd5a2cf9b0f328281280"
server_key_version: "encryptionkey@0"
I20260810 18:09:07.016801 5686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "e55952bf73b34a8ea5c0409283de7ab7"
format_stamp: "Formatted at 2026-08-10 18:09:07 on dist-test-slave-27sr"
server_key: "aac1d8e4bcb25fab835ccc0f4088c6c5"
server_key_iv: "b87129078146cd5a2cf9b0f328281280"
server_key_version: "encryptionkey@0"
I20260810 18:09:07.021724 5686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 18:09:07.024928 5701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:07.026350 5686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:09:07.026515 5686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "e55952bf73b34a8ea5c0409283de7ab7"
format_stamp: "Formatted at 2026-08-10 18:09:07 on dist-test-slave-27sr"
server_key: "aac1d8e4bcb25fab835ccc0f4088c6c5"
server_key_iv: "b87129078146cd5a2cf9b0f328281280"
server_key_version: "encryptionkey@0"
I20260810 18:09:07.026645 5686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:07.049463 5686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:07.052963 5686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:07.053216 5686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:07.053970 5686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:07.054966 5686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:09:07.055022 5686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:07.055104 5686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:09:07.055154 5686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:07.066186 5686 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:34773
I20260810 18:09:07.066210 5831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:34773 every 8 connection(s)
I20260810 18:09:07.067320 5686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:09:07.070451 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 5686
I20260810 18:09:07.070593 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:09:07.070957 22233 external_mini_cluster.cc:1428] Setting key 80ebf2ce96987581a976e6256aa2ecef
Aug 10 18:09:07 dist-test-slave-27sr krb5kdc[5246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385347, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:07.086995 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.069476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:42123 (local address 127.21.182.126:44475)
0810 18:09:07.069626 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:07.069630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:07.070679 (+ 1049us) server_negotiation.cc:407] Connection header received
0810 18:09:07.072779 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:07.072783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:07.072846 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:07.072965 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:07.074473 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.075065 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.076356 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.076619 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.076715 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.081874 (+ 5159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:07.081878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:07.081880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:07.081919 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:07.083634 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:07.084574 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:07.084578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:07.084579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:07.084721 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:07.085525 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:07.085531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:07.085534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:07.085826 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:07.085977 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:07.086709 (+ 732us) server_negotiation.cc:299] Negotiation successful
0810 18:09:07.086838 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:09:07.088023 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.069865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:42123)
0810 18:09:07.070487 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:07.070554 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:07.072485 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:07.073179 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:07.073187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:07.073769 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:07.074306 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.074321 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.075184 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.075186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.076230 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.076241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.076770 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.076773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.076982 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.077728 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:07.077746 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:07.079472 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:07.083832 (+ 4360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:07.083860 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:07.083932 (+ 72us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:07.084437 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:07.085042 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:07.085045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:07.085047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:07.085388 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:07.086001 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:07.086013 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:07.086425 (+ 412us) client_negotiation.cc:769] Sending connection context
0810 18:09:07.086622 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 18:09:07.086979 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":122,"threads_started":1}
I20260810 18:09:07.089591 5832 heartbeater.cc:347] Connected to a master server at 127.21.182.126:44475
I20260810 18:09:07.089871 5832 heartbeater.cc:464] Registering TS with master...
I20260810 18:09:07.090451 5832 heartbeater.cc:511] Master 127.21.182.126:44475 requested a full tablet report, sending...
I20260810 18:09:07.091719 5295 ts_manager.cc:194] Registered new tserver with Master: e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773)
I20260810 18:09:07.092478 5295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:42123
I20260810 18:09:07.098554 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:09:07.108896 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.101107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.21.182.126:44475)
0810 18:09:07.101242 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:07.101246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:07.101464 (+ 218us) server_negotiation.cc:407] Connection header received
0810 18:09:07.101573 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:07.101575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:07.101622 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:07.101698 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:07.102489 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.103011 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.103998 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.104253 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.104355 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.105822 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:07.105826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:07.105828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:07.105858 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:07.107179 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:07.107625 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:07.107627 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:07.107629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:07.107713 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:07.108044 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:07.108047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:07.108048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:07.108309 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:07.108442 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:07.108648 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 18:09:07.108760 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:09:07.112514 5295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:09:07.115738 5295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:09:07.133000 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.125836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34773 (local address 127.0.0.1:60474)
0810 18:09:07.126763 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:07.126823 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:07.127030 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:07.127932 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:07.127939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:07.127978 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:07.128149 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.128157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.129440 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.129446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.131407 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.131418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.132373 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.132377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.132486 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.132525 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 18:09:07.132641 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 18:09:07.132863 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":389,"spinlock_wait_cycles":5760,"thread_start_us":121,"threads_started":1}
I20260810 18:09:07.132977 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.126032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60474 (local address 127.21.182.67:34773)
0810 18:09:07.127364 (+ 1332us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:07.127379 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:07.127405 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:09:07.127482 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:07.127488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:07.127701 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:07.127926 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:07.128273 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.129294 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.131509 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.132238 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.132376 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.132545 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:07.132686 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 18:09:07.132800 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1206,"thread_start_us":199,"threads_started":1}
I20260810 18:09:07.133201 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.126051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:34303 (local address 127.0.0.1:44194)
0810 18:09:07.126900 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:07.126914 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:07.127031 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:07.127932 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:07.127939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:07.127979 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:07.128354 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.128360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.129833 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.129837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.131206 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.131219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.132687 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.132691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.132894 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.132921 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:09:07.132985 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 18:09:07.133043 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":189,"spinlock_wait_cycles":7296,"thread_start_us":106,"threads_started":1}
I20260810 18:09:07.133416 5850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.127812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35762 (local address 127.21.182.65:45561)
0810 18:09:07.128185 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:07.128194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:07.128219 (+ 25us) server_negotiation.cc:407] Connection header received
0810 18:09:07.128298 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:07.128310 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:07.128531 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:07.128687 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:07.128894 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.129970 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.131452 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.132395 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.132490 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.132616 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:07.133133 (+ 517us) server_negotiation.cc:299] Negotiation successful
0810 18:09:07.133251 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":153,"threads_started":1}
I20260810 18:09:07.133662 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.126925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:45561 (local address 127.0.0.1:35762)
0810 18:09:07.127264 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:07.127278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:07.127368 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:07.128680 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:07.128684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:07.128703 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:07.128792 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.128797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.130089 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.130096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.131298 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.131313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.132687 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.132691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.132849 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.132877 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:09:07.132962 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 18:09:07.133043 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":25,"thread_start_us":98,"threads_started":1}
I20260810 18:09:07.133836 5848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.126222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44194 (local address 127.21.182.66:34303)
0810 18:09:07.126979 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:07.126993 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:07.127021 (+ 28us) server_negotiation.cc:407] Connection header received
0810 18:09:07.127222 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:07.127229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:07.127472 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:07.127655 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:07.128482 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.129669 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.131352 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.132322 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.132466 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.132616 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:07.133533 (+ 917us) server_negotiation.cc:299] Negotiation successful
0810 18:09:07.133649 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":122,"threads_started":1}
I20260810 18:09:07.137130 5601 tablet_service.cc:1467] Processing CreateTablet for tablet 80676af56db64886b6752af767702b01 (DEFAULT_TABLE table=test-table [id=0885ff205a51441299500ffcc48ddaaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:09:07.137627 5440 tablet_service.cc:1467] Processing CreateTablet for tablet 80676af56db64886b6752af767702b01 (DEFAULT_TABLE table=test-table [id=0885ff205a51441299500ffcc48ddaaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:09:07.138052 5753 tablet_service.cc:1467] Processing CreateTablet for tablet 80676af56db64886b6752af767702b01 (DEFAULT_TABLE table=test-table [id=0885ff205a51441299500ffcc48ddaaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:09:07.138563 5601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80676af56db64886b6752af767702b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:07.138731 5440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80676af56db64886b6752af767702b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:07.139093 5753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80676af56db64886b6752af767702b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:07.145535 5852 tablet_bootstrap.cc:492] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: Bootstrap starting.
I20260810 18:09:07.147361 5854 tablet_bootstrap.cc:492] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7: Bootstrap starting.
I20260810 18:09:07.148255 5853 tablet_bootstrap.cc:492] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Bootstrap starting.
I20260810 18:09:07.148299 5852 tablet_bootstrap.cc:654] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:07.149273 5852 log.cc:824] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:07.150429 5854 tablet_bootstrap.cc:654] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:07.150843 5853 tablet_bootstrap.cc:654] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:07.151332 5854 log.cc:824] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:07.151742 5853 log.cc:824] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:07.152223 5852 tablet_bootstrap.cc:492] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: No bootstrap required, opened a new log
I20260810 18:09:07.152464 5852 ts_tablet_manager.cc:1397] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:09:07.153679 5854 tablet_bootstrap.cc:492] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7: No bootstrap required, opened a new log
I20260810 18:09:07.153899 5854 ts_tablet_manager.cc:1397] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:09:07.157696 5853 tablet_bootstrap.cc:492] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac: No bootstrap required, opened a new log
I20260810 18:09:07.157997 5853 ts_tablet_manager.cc:1397] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260810 18:09:07.157999 5854 raft_consensus.cc:348] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:07.158345 5854 raft_consensus.cc:374] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:07.158432 5854 raft_consensus.cc:729] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e55952bf73b34a8ea5c0409283de7ab7, State: Initialized, Role: FOLLOWER
I20260810 18:09:07.158797 5852 raft_consensus.cc:348] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:07.159094 5852 raft_consensus.cc:374] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:07.159011 5854 consensus_queue.cc:260] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:07.159160 5852 raft_consensus.cc:729] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c4da52e8d044108d1aff802619c4d5, State: Initialized, Role: FOLLOWER
I20260810 18:09:07.159698 5852 consensus_queue.cc:260] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:07.159822 5832 heartbeater.cc:503] Master 127.21.182.126:44475 was elected leader, sending a full tablet report...
I20260810 18:09:07.160271 5854 ts_tablet_manager.cc:1428] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 18:09:07.160739 5852 ts_tablet_manager.cc:1428] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 18:09:07.160912 5524 heartbeater.cc:503] Master 127.21.182.126:44475 was elected leader, sending a full tablet report...
I20260810 18:09:07.162156 5853 raft_consensus.cc:348] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:07.162472 5853 raft_consensus.cc:374] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:07.162544 5853 raft_consensus.cc:729] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cecfd1fc63434dbb19d83c1d1ce5ac, State: Initialized, Role: FOLLOWER
I20260810 18:09:07.163034 5853 consensus_queue.cc:260] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:07.163846 5681 heartbeater.cc:503] Master 127.21.182.126:44475 was elected leader, sending a full tablet report...
I20260810 18:09:07.164242 5853 ts_tablet_manager.cc:1428] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 18:09:07.193327 5858 raft_consensus.cc:482] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:09:07.193588 5858 raft_consensus.cc:504] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:07.194903 5858 leader_election.cc:288] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303), 70c4da52e8d044108d1aff802619c4d5 (127.21.182.65:45561)
I20260810 18:09:07.199995 5850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.195406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:41087 (local address 127.21.182.65:45561)
0810 18:09:07.195576 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:07.195581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:07.195818 (+ 237us) server_negotiation.cc:407] Connection header received
0810 18:09:07.196007 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:07.196009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:07.196069 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:07.196153 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:07.196410 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.197121 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.198516 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.199194 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.199305 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.199366 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:07.199727 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 18:09:07.199841 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:09:07.199994 5848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.195313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:47441 (local address 127.21.182.66:34303)
0810 18:09:07.195468 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:07.195475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:07.195493 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:09:07.195558 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:07.195561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:07.195625 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:07.195707 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:07.196121 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.196913 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.198511 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.199157 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.199323 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.199368 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:07.199754 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 18:09:07.199827 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:09:07.200028 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.195144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:34303 (local address 127.21.182.67:47441)
0810 18:09:07.195298 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:07.195316 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:07.195420 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:07.195753 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:07.195757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:07.195787 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:07.195990 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.196001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.197033 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.197036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.198320 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.198333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.199304 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.199309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.199549 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.199591 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 18:09:07.199695 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 18:09:07.199834 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":73}
I20260810 18:09:07.200038 5861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.195161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:45561 (local address 127.21.182.67:41087)
0810 18:09:07.195732 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:07.195745 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:07.195847 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:07.196177 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:07.196179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:07.196196 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:07.196296 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.196300 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.197223 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.197226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.198396 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.198406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.199304 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.199309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.199545 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.199595 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 18:09:07.199682 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 18:09:07.199906 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":266,"spinlock_wait_cycles":1792,"thread_start_us":153,"threads_started":1}
I20260810 18:09:07.200913 5625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80676af56db64886b6752af767702b01" candidate_uuid: "e55952bf73b34a8ea5c0409283de7ab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" is_pre_election: true
I20260810 18:09:07.200939 5466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80676af56db64886b6752af767702b01" candidate_uuid: "e55952bf73b34a8ea5c0409283de7ab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c4da52e8d044108d1aff802619c4d5" is_pre_election: true
I20260810 18:09:07.201280 5466 raft_consensus.cc:2454] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e55952bf73b34a8ea5c0409283de7ab7 in term 0.
I20260810 18:09:07.201280 5625 raft_consensus.cc:2454] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e55952bf73b34a8ea5c0409283de7ab7 in term 0.
I20260810 18:09:07.201848 5705 leader_election.cc:302] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07cecfd1fc63434dbb19d83c1d1ce5ac, 70c4da52e8d044108d1aff802619c4d5, e55952bf73b34a8ea5c0409283de7ab7; no voters:
I20260810 18:09:07.202190 5858 raft_consensus.cc:2781] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:09:07.202280 5858 raft_consensus.cc:482] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:09:07.202312 5858 raft_consensus.cc:3037] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:07.203462 5858 raft_consensus.cc:504] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:07.203861 5858 leader_election.cc:288] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [CANDIDATE]: Term 1 election: Requested vote from peers 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303), 70c4da52e8d044108d1aff802619c4d5 (127.21.182.65:45561)
I20260810 18:09:07.204367 5625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80676af56db64886b6752af767702b01" candidate_uuid: "e55952bf73b34a8ea5c0409283de7ab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac"
I20260810 18:09:07.204367 5466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80676af56db64886b6752af767702b01" candidate_uuid: "e55952bf73b34a8ea5c0409283de7ab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c4da52e8d044108d1aff802619c4d5"
I20260810 18:09:07.204524 5625 raft_consensus.cc:3037] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:07.204524 5466 raft_consensus.cc:3037] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:07.206138 5466 raft_consensus.cc:2454] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e55952bf73b34a8ea5c0409283de7ab7 in term 1.
I20260810 18:09:07.206138 5625 raft_consensus.cc:2454] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e55952bf73b34a8ea5c0409283de7ab7 in term 1.
I20260810 18:09:07.206563 5703 leader_election.cc:302] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70c4da52e8d044108d1aff802619c4d5, e55952bf73b34a8ea5c0409283de7ab7; no voters:
I20260810 18:09:07.206802 5858 raft_consensus.cc:2781] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:09:07.207108 5858 raft_consensus.cc:686] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [term 1 LEADER]: Becoming Leader. State: Replica: e55952bf73b34a8ea5c0409283de7ab7, State: Running, Role: LEADER
I20260810 18:09:07.207532 5858 consensus_queue.cc:237] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:07.211193 5294 catalog_manager.cc:5495] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67). New cstate: current_term: 1 leader_uuid: "e55952bf73b34a8ea5c0409283de7ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } health_report { overall_health: UNKNOWN } } }
I20260810 18:09:07.237231 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.233255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.21.182.67:34773)
0810 18:09:07.233438 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:07.233443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:07.233459 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:09:07.233648 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:07.233651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:07.233712 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:07.233836 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:09:07.234153 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.234683 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.235575 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.235804 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.235944 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.236310 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:09:07.236786 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:09:07.236934 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:07.236980 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 18:09:07.237045 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 18:09:07.244015 5625 raft_consensus.cc:1261] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 1 FOLLOWER]: Refusing update from remote peer e55952bf73b34a8ea5c0409283de7ab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:09:07.244036 5466 raft_consensus.cc:1261] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Refusing update from remote peer e55952bf73b34a8ea5c0409283de7ab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:09:07.244879 5862 consensus_queue.cc:1035] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:09:07.245113 5858 consensus_queue.cc:1035] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:09:07.255762 5867 mvcc.cc:205] Tried to move back new op lower bound from 7317034382302932992 to 7317034382165606400. Current Snapshot: MvccSnapshot[applied={T|T < 7317034382302932992}]
I20260810 18:09:07.255761 5868 mvcc.cc:205] Tried to move back new op lower bound from 7317034382302932992 to 7317034382165606400. Current Snapshot: MvccSnapshot[applied={T|T < 7317034382302932992}]
I20260810 18:09:07.257125 5866 mvcc.cc:205] Tried to move back new op lower bound from 7317034382302932992 to 7317034382165606400. Current Snapshot: MvccSnapshot[applied={T|T < 7317034382302932992}]
I20260810 18:09:07.267681 5293 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:09:07.267926 5293 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:09:07.270967 5293 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Sending DeleteTablet for 3 replicas of tablet 80676af56db64886b6752af767702b01
I20260810 18:09:07.272099 5753 tablet_service.cc:1514] Processing DeleteTablet for tablet 80676af56db64886b6752af767702b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:09:07 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:60474
I20260810 18:09:07.272104 5601 tablet_service.cc:1514] Processing DeleteTablet for tablet 80676af56db64886b6752af767702b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:09:07 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:44194
I20260810 18:09:07.272095 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet 80676af56db64886b6752af767702b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:09:07 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:35762
I20260810 18:09:07.272670 5878 tablet_replica.cc:330] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac: stopping tablet replica
I20260810 18:09:07.272965 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 5365
I20260810 18:09:07.283222 5878 raft_consensus.cc:2229] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:09:07.283690 5881 tablet_replica.cc:330] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7: stopping tablet replica
I20260810 18:09:07.283850 5878 raft_consensus.cc:2258] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:09:07.284178 5881 raft_consensus.cc:2229] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:09:07.284759 5881 raft_consensus.cc:2258] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:09:07.286602 5878 ts_tablet_manager.cc:1905] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:09:07.287010 5881 ts_tablet_manager.cc:1905] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:09:07.289855 5878 ts_tablet_manager.cc:1918] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:09:07.290086 5878 log.cc:1196] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/80676af56db64886b6752af767702b01
I20260810 18:09:07.290524 5878 ts_tablet_manager.cc:1939] T 80676af56db64886b6752af767702b01 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Deleting consensus metadata
I20260810 18:09:07.290845 5881 ts_tablet_manager.cc:1918] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:09:07.290989 5881 log.cc:1196] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/80676af56db64886b6752af767702b01
I20260810 18:09:07.291355 5881 ts_tablet_manager.cc:1939] T 80676af56db64886b6752af767702b01 P e55952bf73b34a8ea5c0409283de7ab7: Deleting consensus metadata
I20260810 18:09:07.292713 5281 catalog_manager.cc:4841] TS 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303): tablet 80676af56db64886b6752af767702b01 (table test-table [id=0885ff205a51441299500ffcc48ddaaf]) successfully deleted
I20260810 18:09:07.292716 5278 catalog_manager.cc:4841] TS e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773): tablet 80676af56db64886b6752af767702b01 (table test-table [id=0885ff205a51441299500ffcc48ddaaf]) successfully deleted
I20260810 18:09:07.293234 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 5529
W20260810 18:09:07.293206 5278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:45561 (error 108)
I20260810 18:09:07.306228 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.305877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:45561 (local address 127.0.0.1:35768)
0810 18:09:07.306036 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:07.306133 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 18:09:07.306491 5278 catalog_manager.cc:4584] TS 70c4da52e8d044108d1aff802619c4d5 (127.21.182.65:45561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80676af56db64886b6752af767702b01: Network error: Client connection negotiation failed: client connection to 127.21.182.65:45561: connect: Connection refused (error 111)
I20260810 18:09:07.312693 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 5686
I20260810 18:09:07.322659 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 5260
I20260810 18:09:07.334895 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:44475
--webserver_interface=127.21.182.126
--webserver_port=38949
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:44475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:09:07.467707 5883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:07.468071 5883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:07.468118 5883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:07.472239 5883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:09:07.472313 5883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:07.472383 5883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:07.472414 5883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:09:07.472435 5883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:09:07.478199 5883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:44475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:44475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=38949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:07.479429 5883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:07.480566 5883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:09:07.497889 5883 server_base.cc:1034] running on GCE node
W20260810 18:09:07.497782 5889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:07.497809 5890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:07.497782 5892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:07.498809 5883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:07.500337 5883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:07.501529 5883 hybrid_clock.cc:648] HybridClock initialized: now 1786385347501508 us; error 36 us; skew 500 ppm
Aug 10 18:09:07 dist-test-slave-27sr krb5kdc[5246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385347, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:07.504676 5883 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:09:07.505766 5883 webserver.cc:460] Webserver started at http://127.21.182.126:38949/ using document root <none> and password file <none>
I20260810 18:09:07.506435 5883 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:07.506561 5883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:07.511015 5883 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:09:07.513629 5899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:07.515020 5883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:09:07.515269 5883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "3e8fd4fcef724555a349913e33c10286"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "0d7047b7bd5f6f686bb56aeb696da719"
server_key_iv: "9241dc5ce1bd13fd817d0042be058a0c"
server_key_version: "encryptionkey@0"
I20260810 18:09:07.516122 5883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:07.530462 5883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:07.534296 5883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:07.534543 5883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:07.542721 5883 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:44475
I20260810 18:09:07.543038 5959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:44475 every 8 connection(s)
I20260810 18:09:07.544095 5883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:09:07.549049 5960 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:09:07.551056 5960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Bootstrap starting.
I20260810 18:09:07.554049 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 5883
I20260810 18:09:07.554636 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:45561
--local_ip_for_outbound_sockets=127.21.182.65
--tserver_master_addrs=127.21.182.126:44475
--webserver_port=38689
--webserver_interface=127.21.182.65
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:09:07.570894 5960 log.cc:824] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:07.581056 5960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:09:07.581532 5960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Bootstrap complete.
I20260810 18:09:07.586068 5960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } }
I20260810 18:09:07.586573 5960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8fd4fcef724555a349913e33c10286, State: Initialized, Role: FOLLOWER
I20260810 18:09:07.587131 5960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } }
I20260810 18:09:07.587287 5960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:09:07.587399 5960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:09:07.587522 5960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:09:07.589753 5960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } }
I20260810 18:09:07.590193 5960 leader_election.cc:302] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e8fd4fcef724555a349913e33c10286; no voters:
I20260810 18:09:07.590613 5960 leader_election.cc:288] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:09:07.590711 5964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:09:07.590922 5964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 2 LEADER]: Becoming Leader. State: Replica: 3e8fd4fcef724555a349913e33c10286, State: Running, Role: LEADER
I20260810 18:09:07.591261 5964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } }
I20260810 18:09:07.592157 5960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:09:07.592870 5965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3e8fd4fcef724555a349913e33c10286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } } }
I20260810 18:09:07.593035 5965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:07.593385 5966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e8fd4fcef724555a349913e33c10286. Latest consensus state: current_term: 2 leader_uuid: "3e8fd4fcef724555a349913e33c10286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } } }
I20260810 18:09:07.593556 5966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:07.593874 5972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:09:07.599411 5972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:09:07.600361 5972 catalog_manager.cc:1259] Loaded cluster ID: e063ee4ac5624bb48ccc37b3601d0a06
I20260810 18:09:07.600437 5972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:09:07.601965 5972 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 18:09:07.606671 5972 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 18:09:07.613377 5972 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:09:07.614321 5972 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Loaded TSK: 0
I20260810 18:09:07.615695 5972 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:09:07.687508 5962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:07.687842 5962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:07.687942 5962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:07.692305 5962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:07.692427 5962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:07.692538 5962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:09:07.698663 5962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:45561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=38689
--tserver_master_addrs=127.21.182.126:44475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:07.700109 5962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:07.701349 5962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:07.712157 5994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:07.712314 5962 server_base.cc:1034] running on GCE node
W20260810 18:09:07.712155 5991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:07.712147 5992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:07.712973 5962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:07.713814 5962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:07.715019 5962 hybrid_clock.cc:648] HybridClock initialized: now 1786385347714990 us; error 57 us; skew 500 ppm
Aug 10 18:09:07 dist-test-slave-27sr krb5kdc[5246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385347, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:07.718219 5962 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:09:07.719432 5962 webserver.cc:460] Webserver started at http://127.21.182.65:38689/ using document root <none> and password file <none>
I20260810 18:09:07.720134 5962 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:07.720260 5962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:07.724732 5962 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:07.727238 6001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:07.728884 5962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:09:07.729034 5962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "70c4da52e8d044108d1aff802619c4d5"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "fd1e782ae19e84525ea4f3e24c6b2c86"
server_key_iv: "9ae3a24352f62a47a50bc881cbed13b2"
server_key_version: "encryptionkey@0"
I20260810 18:09:07.729764 5962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:07.745697 5962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:07.749547 5962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:07.749788 5962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:07.750545 5962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:07.751801 6008 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 18:09:07.756417 5962 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 18:09:07.756505 5962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 18:09:07.756601 5962 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 18:09:07.758570 5962 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 18:09:07.758625 5962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 18:09:07.758989 6008 tablet_bootstrap.cc:492] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: Bootstrap starting.
I20260810 18:09:07.770337 5962 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:45561
I20260810 18:09:07.770413 6128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:45561 every 8 connection(s)
I20260810 18:09:07.771514 5962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:09:07.774039 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 5962
I20260810 18:09:07.774535 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:34303
--local_ip_for_outbound_sockets=127.21.182.66
--tserver_master_addrs=127.21.182.126:44475
--webserver_port=34321
--webserver_interface=127.21.182.66
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:09:07.779012 6008 log.cc:824] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:09:07 dist-test-slave-27sr krb5kdc[5246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385347, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:07.788107 6008 tablet_bootstrap.cc:492] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:09:07.788597 6008 tablet_bootstrap.cc:492] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: Bootstrap complete.
I20260810 18:09:07.789956 6008 ts_tablet_manager.cc:1397] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.004s
I20260810 18:09:07.793160 6008 raft_consensus.cc:348] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:07.793814 6008 raft_consensus.cc:729] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c4da52e8d044108d1aff802619c4d5, State: Initialized, Role: FOLLOWER
I20260810 18:09:07.794414 6008 consensus_queue.cc:260] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:07.794682 6132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.773929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:58207 (local address 127.21.182.126:44475)
0810 18:09:07.774455 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:07.774476 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:07.775108 (+ 632us) server_negotiation.cc:407] Connection header received
0810 18:09:07.776937 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:07.776957 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:07.777441 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:07.777882 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:07.779458 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.781181 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.782923 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.783617 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.783808 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.787510 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:07.787520 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:07.787544 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:07.787628 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:07.790369 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:07.791130 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:07.791138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:07.791146 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:07.791328 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:07.791884 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:07.791907 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:07.791909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:07.792442 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:07.792590 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:07.793288 (+ 698us) server_negotiation.cc:299] Negotiation successful
0810 18:09:07.793654 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":131,"threads_started":1}
I20260810 18:09:07.795198 6131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.773771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:58207)
0810 18:09:07.774869 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:07.774943 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:07.776274 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:07.777957 (+ 1683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:07.777966 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:07.778517 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:07.779235 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.779257 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.781380 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.781384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.782734 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.782746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.783822 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.783830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.784125 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.784939 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:07.784966 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:07.787210 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:07.790562 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:07.790569 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:07.790586 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:07.790981 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:07.791448 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:07.791452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:07.791454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:07.791651 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:07.792586 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:07.792594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:07.792927 (+ 333us) client_negotiation.cc:769] Sending connection context
0810 18:09:07.793143 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 18:09:07.793523 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":161,"threads_started":1}
I20260810 18:09:07.795552 6008 ts_tablet_manager.cc:1428] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:09:07.797757 6129 heartbeater.cc:347] Connected to a master server at 127.21.182.126:44475
I20260810 18:09:07.798002 6129 heartbeater.cc:464] Registering TS with master...
I20260810 18:09:07.798529 6129 heartbeater.cc:511] Master 127.21.182.126:44475 requested a full tablet report, sending...
I20260810 18:09:07.801429 5917 ts_manager.cc:194] Registered new tserver with Master: 70c4da52e8d044108d1aff802619c4d5 (127.21.182.65:45561)
I20260810 18:09:07.805295 5917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:58207
I20260810 18:09:07.808192 6129 heartbeater.cc:503] Master 127.21.182.126:44475 was elected leader, sending a full tablet report...
Aug 10 18:09:07 dist-test-slave-27sr krb5kdc[5246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385347, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for kudu/127.21.182.65@KRBTEST.COM
I20260810 18:09:07.816422 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.804799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:45561 (local address 127.0.0.1:35780)
0810 18:09:07.805678 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:07.805723 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:07.805884 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:07.806445 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:07.806452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:07.806932 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:07.807073 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.807082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.808506 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.808510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.809313 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.809323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.809956 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.809960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.810081 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.810733 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:07.810748 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:07.812296 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:07.814280 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:07.814292 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:07.814299 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:07.814566 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:07.814987 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:07.814990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:07.814992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:07.815091 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:07.815617 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:07.815624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:07.815769 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 18:09:07.815956 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 18:09:07.816217 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":118,"threads_started":1}
I20260810 18:09:07.816422 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.804909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35780 (local address 127.21.182.65:45561)
0810 18:09:07.805398 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:07.805417 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:07.805855 (+ 438us) server_negotiation.cc:407] Connection header received
0810 18:09:07.806048 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:07.806054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:07.806268 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:07.806413 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:07.807218 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.808373 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.809475 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.809816 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.809951 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.812468 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:07.812478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:07.812490 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:07.812515 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:07.814144 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:07.814715 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:07.814721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:07.814723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:07.814811 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:07.815201 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:07.815204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:07.815206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:07.815491 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:07.815646 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:07.816047 (+ 401us) server_negotiation.cc:299] Negotiation successful
0810 18:09:07.816248 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":132,"threads_started":1}
I20260810 18:09:07.817637 6054 tablet_service.cc:1514] Processing DeleteTablet for tablet 80676af56db64886b6752af767702b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:09:07 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:35780
I20260810 18:09:07.818758 6138 tablet_replica.cc:330] stopping tablet replica
I20260810 18:09:07.819110 6138 raft_consensus.cc:2229] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:09:07.819437 6138 raft_consensus.cc:2258] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:09:07.820652 6138 ts_tablet_manager.cc:1905] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:09:07.824735 6138 ts_tablet_manager.cc:1918] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:09:07.824877 6138 log.cc:1196] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal/wals/80676af56db64886b6752af767702b01
I20260810 18:09:07.825179 6138 ts_tablet_manager.cc:1939] T 80676af56db64886b6752af767702b01 P 70c4da52e8d044108d1aff802619c4d5: Deleting consensus metadata
I20260810 18:09:07.827255 5901 catalog_manager.cc:4841] TS 70c4da52e8d044108d1aff802619c4d5 (127.21.182.65:45561): tablet 80676af56db64886b6752af767702b01 (table test-table [id=0885ff205a51441299500ffcc48ddaaf]) successfully deleted
W20260810 18:09:07.910933 6133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:07.911299 6133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:07.911397 6133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:07.916049 6133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:07.916127 6133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:07.916193 6133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:09:07.922156 6133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:34303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=34321
--tserver_master_addrs=127.21.182.126:44475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:07.923416 6133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:07.924705 6133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:07.935484 6146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:07.935437 6144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:07.935424 6143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:07.936074 6133 server_base.cc:1034] running on GCE node
I20260810 18:09:07.936570 6133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:07.937302 6133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:07.938509 6133 hybrid_clock.cc:648] HybridClock initialized: now 1786385347938478 us; error 46 us; skew 500 ppm
Aug 10 18:09:07 dist-test-slave-27sr krb5kdc[5246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385347, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:07.941697 6133 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:09:07.942780 6133 webserver.cc:460] Webserver started at http://127.21.182.66:34321/ using document root <none> and password file <none>
I20260810 18:09:07.943424 6133 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:07.943547 6133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:07.947834 6133 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:07.950248 6154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:07.951574 6133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:09:07.951719 6133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "26839255bc1c4be8ab44132c335dd72d"
server_key_iv: "ecb26a6e7cfeba55d6123c3c7d8c6742"
server_key_version: "encryptionkey@0"
I20260810 18:09:07.952365 6133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:07.969038 6133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:07.972698 6133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:07.972929 6133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:07.973577 6133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:07.974468 6133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:09:07.974540 6133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:07.974611 6133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:09:07.974663 6133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:07.984578 6133 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:34303
I20260810 18:09:07.984601 6277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:34303 every 8 connection(s)
I20260810 18:09:07.985781 6133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:09:07.995694 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 6133
I20260810 18:09:07.996212 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:34773
--local_ip_for_outbound_sockets=127.21.182.67
--tserver_master_addrs=127.21.182.126:44475
--webserver_port=35299
--webserver_interface=127.21.182.67
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:09:07 dist-test-slave-27sr krb5kdc[5246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385347, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:08.001995 6132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.987948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:33639 (local address 127.21.182.126:44475)
0810 18:09:07.988103 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:07.988107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:07.989002 (+ 895us) server_negotiation.cc:407] Connection header received
0810 18:09:07.990495 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:07.990498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:07.990554 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:07.990634 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:07.991985 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.992609 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.994005 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.994272 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.994367 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.997270 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:07.997273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:07.997275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:07.997305 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:07.999305 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:07.999962 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:07.999969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:07.999973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:08.000083 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:08.000544 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:08.000551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:08.000556 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:08.000854 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:08.000980 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:08.001545 (+ 565us) server_negotiation.cc:299] Negotiation successful
0810 18:09:08.001724 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:09:08.002837 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:07.988308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:33639)
0810 18:09:07.988821 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:07.988869 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:07.990226 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:07.990827 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:07.990834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:07.991254 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:07.991775 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.991789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.992749 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.992752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.993883 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:07.993895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:07.994447 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:07.994450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:07.994610 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:07.995198 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:07.995215 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:07.997060 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:07.999462 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:07.999473 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:07.999495 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:07.999809 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:08.000224 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:08.000229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:08.000231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:08.000435 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:08.000977 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:08.000983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:08.001257 (+ 274us) client_negotiation.cc:769] Sending connection context
0810 18:09:08.001441 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 18:09:08.001782 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":119,"threads_started":1}
I20260810 18:09:08.004243 6278 heartbeater.cc:347] Connected to a master server at 127.21.182.126:44475
I20260810 18:09:08.004529 6278 heartbeater.cc:464] Registering TS with master...
I20260810 18:09:08.005110 6278 heartbeater.cc:511] Master 127.21.182.126:44475 requested a full tablet report, sending...
I20260810 18:09:08.006338 5917 ts_manager.cc:194] Registered new tserver with Master: 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303)
I20260810 18:09:08.006959 5917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:33639
W20260810 18:09:08.124701 6281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:08.125036 6281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:08.125130 6281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:08.130002 6281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:08.130097 6281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:08.130240 6281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:09:08.136905 6281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:34773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=35299
--tserver_master_addrs=127.21.182.126:44475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:08.138201 6281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:08.139339 6281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:08.150318 6286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:08.150364 6281 server_base.cc:1034] running on GCE node
W20260810 18:09:08.150323 6287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:08.150323 6289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:08.151026 6281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:08.151770 6281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:08.152979 6281 hybrid_clock.cc:648] HybridClock initialized: now 1786385348152959 us; error 33 us; skew 500 ppm
Aug 10 18:09:08 dist-test-slave-27sr krb5kdc[5246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385348, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:08.156153 6281 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:09:08.157204 6281 webserver.cc:460] Webserver started at http://127.21.182.67:35299/ using document root <none> and password file <none>
I20260810 18:09:08.157855 6281 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:08.157958 6281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:08.162396 6281 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:09:08.164798 6297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:08.166177 6281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:09:08.166326 6281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "e55952bf73b34a8ea5c0409283de7ab7"
format_stamp: "Formatted at 2026-08-10 18:09:07 on dist-test-slave-27sr"
server_key: "aac1d8e4bcb25fab835ccc0f4088c6c5"
server_key_iv: "b87129078146cd5a2cf9b0f328281280"
server_key_version: "encryptionkey@0"
I20260810 18:09:08.166947 6281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:08.180513 6281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:08.184571 6281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:08.184821 6281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:08.185495 6281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:08.186465 6281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:09:08.186543 6281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:08.186618 6281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:09:08.186671 6281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:08.196810 6281 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:34773
I20260810 18:09:08.196874 6424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:34773 every 8 connection(s)
I20260810 18:09:08.198011 6281 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:09:08.205665 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 6281
Aug 10 18:09:08 dist-test-slave-27sr krb5kdc[5246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385348, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:08.214393 6132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.200260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:56935 (local address 127.21.182.126:44475)
0810 18:09:08.200438 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:08.200443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:08.201517 (+ 1074us) server_negotiation.cc:407] Connection header received
0810 18:09:08.202941 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:08.202944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:08.203002 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:08.203086 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:08.204494 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.205052 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.206708 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.207055 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.207136 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:08.210145 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:08.210148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:08.210150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:08.210176 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:08.211882 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:08.212671 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:08.212674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:08.212675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:08.212757 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:08.213208 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:08.213210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:08.213212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:08.213394 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:08.213508 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:08.214085 (+ 577us) server_negotiation.cc:299] Negotiation successful
0810 18:09:08.214219 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:09:08.215230 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.206494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50078 (local address 127.21.182.126:44475)
0810 18:09:08.206846 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:08.206850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:08.206864 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:09:08.206978 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:08.206981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:08.207033 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:08.207132 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:08.207918 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.208515 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.209671 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.209920 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.210035 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:08.211418 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:08.211422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:08.211425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:08.211472 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:08.213211 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:08.213654 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:08.213657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:08.213659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:08.213749 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:08.214043 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:08.214046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:08.214048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:08.214236 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:08.214476 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:08.214941 (+ 465us) server_negotiation.cc:299] Negotiation successful
0810 18:09:08.215072 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":114,"threads_started":1}
I20260810 18:09:08.215593 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.200783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:56935)
0810 18:09:08.201325 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:08.201372 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:08.202699 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:08.203308 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:08.203315 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:08.203706 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:08.204344 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:08.204358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.205190 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.205193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:08.206553 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:08.206564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.207342 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.207345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:08.207495 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:08.208195 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:08.208213 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:08.209949 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:08.212057 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:08.212067 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:08.212090 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:08.212559 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:08.212879 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:08.212883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:08.212886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:08.213103 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:08.213521 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:08.213530 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:08.213819 (+ 289us) client_negotiation.cc:769] Sending connection context
0810 18:09:08.214033 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 18:09:08.214453 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":127,"threads_started":1}
I20260810 18:09:08.217032 6425 heartbeater.cc:347] Connected to a master server at 127.21.182.126:44475
I20260810 18:09:08.217316 6425 heartbeater.cc:464] Registering TS with master...
I20260810 18:09:08.217960 6425 heartbeater.cc:511] Master 127.21.182.126:44475 requested a full tablet report, sending...
I20260810 18:09:08.219010 5917 ts_manager.cc:194] Registered new tserver with Master: e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773)
I20260810 18:09:08.219607 5917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:56935
I20260810 18:09:08.229776 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:09:08.234591 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.230470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.21.182.126:44475)
0810 18:09:08.230674 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:08.230678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:08.230785 (+ 107us) server_negotiation.cc:407] Connection header received
0810 18:09:08.230957 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:08.230960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:08.231074 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:08.231172 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:09:08.231400 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.232050 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.232946 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.233198 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.233324 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:08.233716 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:09:08.234162 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:09:08.234306 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:08.234366 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 18:09:08.234432 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260810 18:09:08.235304 5917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:09:08.239005 5917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:09:08.248391 6054 tablet_service.cc:1467] Processing CreateTablet for tablet 27f5622f1452472a9c43988f6f782263 (DEFAULT_TABLE table=test-table [id=0446a97265204ff297f11af44e8a8cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:09:08.249184 6054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f5622f1452472a9c43988f6f782263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:08.252087 6008 tablet_bootstrap.cc:492] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5: Bootstrap starting.
I20260810 18:09:08.253031 6008 tablet_bootstrap.cc:654] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:08.253818 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.247123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:34303 (local address 127.0.0.1:44196)
0810 18:09:08.247315 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:08.247332 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:08.247427 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:08.248623 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:08.248627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:08.248648 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:08.248786 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:08.248796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.250618 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.250622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:08.252016 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:08.252029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.253177 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.253181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:08.253364 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:08.253431 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 18:09:08.253516 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 18:09:08.253610 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":61}
I20260810 18:09:08.253891 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.247568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60484 (local address 127.21.182.67:34773)
0810 18:09:08.248173 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:08.248186 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:08.248215 (+ 29us) server_negotiation.cc:407] Connection header received
0810 18:09:08.248290 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:08.248295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:08.248463 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:08.248639 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:08.249088 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.250441 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.252212 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.252928 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.253073 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:08.253252 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:08.253601 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 18:09:08.253738 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":96,"threads_started":1}
I20260810 18:09:08.253826 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.247357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34773 (local address 127.0.0.1:60484)
0810 18:09:08.247639 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:08.247664 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:08.247756 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:08.248824 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:08.248830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:08.248853 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:08.248947 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:08.248952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.250655 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.250659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:08.252049 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:08.252061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.253196 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.253203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:08.253400 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:08.253438 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 18:09:08.253554 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 18:09:08.253638 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":122,"thread_start_us":125,"threads_started":1}
I20260810 18:09:08.254473 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.247338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.21.182.66:34303)
0810 18:09:08.247951 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:08.247969 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:08.248002 (+ 33us) server_negotiation.cc:407] Connection header received
0810 18:09:08.248100 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:08.248106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:08.248390 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:08.248971 (+ 581us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:08.249175 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.250456 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.252176 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.253028 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.253993 (+ 965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:08.254156 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:08.254211 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 18:09:08.254321 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":186,"threads_started":1}
I20260810 18:09:08.255398 6008 tablet_bootstrap.cc:492] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5: No bootstrap required, opened a new log
I20260810 18:09:08.255519 6008 ts_tablet_manager.cc:1397] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 18:09:08.256067 6008 raft_consensus.cc:348] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:08.256194 6008 raft_consensus.cc:374] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:08.256244 6008 raft_consensus.cc:729] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c4da52e8d044108d1aff802619c4d5, State: Initialized, Role: FOLLOWER
I20260810 18:09:08.256618 6008 consensus_queue.cc:260] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:08.257550 6008 ts_tablet_manager.cc:1428] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:09:08.258074 6349 tablet_service.cc:1467] Processing CreateTablet for tablet 27f5622f1452472a9c43988f6f782263 (DEFAULT_TABLE table=test-table [id=0446a97265204ff297f11af44e8a8cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:09:08.258066 6205 tablet_service.cc:1467] Processing CreateTablet for tablet 27f5622f1452472a9c43988f6f782263 (DEFAULT_TABLE table=test-table [id=0446a97265204ff297f11af44e8a8cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:09:08.259454 6205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f5622f1452472a9c43988f6f782263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:08.259462 6349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f5622f1452472a9c43988f6f782263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:08.269510 6442 tablet_bootstrap.cc:492] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Bootstrap starting.
I20260810 18:09:08.269510 6441 tablet_bootstrap.cc:492] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7: Bootstrap starting.
I20260810 18:09:08.272339 6441 tablet_bootstrap.cc:654] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:08.272832 6442 tablet_bootstrap.cc:654] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Neither blocks nor log segments found. Creating new log.
W20260810 18:09:08.273016 6130 tablet.cc:2369] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:09:08.273909 6441 log.cc:824] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:08.273909 6442 log.cc:824] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:08.276122 6442 tablet_bootstrap.cc:492] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac: No bootstrap required, opened a new log
I20260810 18:09:08.276356 6442 ts_tablet_manager.cc:1397] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:09:08.277401 6441 tablet_bootstrap.cc:492] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7: No bootstrap required, opened a new log
I20260810 18:09:08.277601 6441 ts_tablet_manager.cc:1397] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 18:09:08.280654 6442 raft_consensus.cc:348] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:08.280993 6442 raft_consensus.cc:374] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:08.281059 6442 raft_consensus.cc:729] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cecfd1fc63434dbb19d83c1d1ce5ac, State: Initialized, Role: FOLLOWER
I20260810 18:09:08.281589 6442 consensus_queue.cc:260] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:08.281580 6441 raft_consensus.cc:348] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:08.281949 6441 raft_consensus.cc:374] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:08.282015 6441 raft_consensus.cc:729] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e55952bf73b34a8ea5c0409283de7ab7, State: Initialized, Role: FOLLOWER
I20260810 18:09:08.282467 6441 consensus_queue.cc:260] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:08.282845 6442 ts_tablet_manager.cc:1428] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:09:08.283259 6278 heartbeater.cc:503] Master 127.21.182.126:44475 was elected leader, sending a full tablet report...
I20260810 18:09:08.283488 6441 ts_tablet_manager.cc:1428] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:09:08.283782 6425 heartbeater.cc:503] Master 127.21.182.126:44475 was elected leader, sending a full tablet report...
I20260810 18:09:08.312661 6135 raft_consensus.cc:482] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:09:08.312987 6135 raft_consensus.cc:504] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:08.314260 6135 leader_election.cc:288] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303), e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773)
I20260810 18:09:08.318503 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.314694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:37643 (local address 127.21.182.66:34303)
0810 18:09:08.314839 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:08.314846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:08.315126 (+ 280us) server_negotiation.cc:407] Connection header received
0810 18:09:08.315288 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:08.315295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:08.315342 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:08.315423 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:08.315724 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.316283 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.317463 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.317860 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.317994 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:08.318053 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:08.318260 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 18:09:08.318344 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:09:08.318504 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.314785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:49247 (local address 127.21.182.67:34773)
0810 18:09:08.314943 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:08.314949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:08.314971 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:09:08.315196 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:08.315199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:08.315242 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:08.315329 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:08.315620 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.316214 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.317489 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.317902 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.318034 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:08.318085 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:08.318267 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 18:09:08.318354 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:09:08.318523 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.314508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34773 (local address 127.21.182.65:49247)
0810 18:09:08.314899 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:08.314916 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:08.315046 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:08.315354 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:08.315357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:08.315375 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:08.315506 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:08.315511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.316335 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.316339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:08.317362 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:08.317372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.318026 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.318030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:08.318126 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:08.318155 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 18:09:08.318238 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 18:09:08.318378 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":90,"thread_start_us":96,"threads_started":1}
I20260810 18:09:08.318532 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.314508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:34303 (local address 127.21.182.65:37643)
0810 18:09:08.315011 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:08.315023 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:08.315163 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:08.315451 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:08.315454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:08.315471 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:08.315599 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:08.315604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.316405 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.316409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:08.317328 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:08.317338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.317994 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.317998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:08.318096 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:08.318150 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 18:09:08.318242 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 18:09:08.318360 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":92,"thread_start_us":67,"threads_started":1}
I20260810 18:09:08.319270 6370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f5622f1452472a9c43988f6f782263" candidate_uuid: "70c4da52e8d044108d1aff802619c4d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e55952bf73b34a8ea5c0409283de7ab7" is_pre_election: true
I20260810 18:09:08.319639 6370 raft_consensus.cc:2454] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70c4da52e8d044108d1aff802619c4d5 in term 0.
I20260810 18:09:08.319657 6229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f5622f1452472a9c43988f6f782263" candidate_uuid: "70c4da52e8d044108d1aff802619c4d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" is_pre_election: true
I20260810 18:09:08.320048 6229 raft_consensus.cc:2454] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70c4da52e8d044108d1aff802619c4d5 in term 0.
I20260810 18:09:08.320243 6003 leader_election.cc:302] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70c4da52e8d044108d1aff802619c4d5, e55952bf73b34a8ea5c0409283de7ab7; no voters:
I20260810 18:09:08.320586 6135 raft_consensus.cc:2781] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:09:08.320678 6135 raft_consensus.cc:482] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:09:08.320730 6135 raft_consensus.cc:3037] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:08.321870 6135 raft_consensus.cc:504] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:08.322240 6135 leader_election.cc:288] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 1 election: Requested vote from peers 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303), e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773)
I20260810 18:09:08.322633 6229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f5622f1452472a9c43988f6f782263" candidate_uuid: "70c4da52e8d044108d1aff802619c4d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac"
I20260810 18:09:08.322806 6229 raft_consensus.cc:3037] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:08.323966 6229 raft_consensus.cc:2454] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70c4da52e8d044108d1aff802619c4d5 in term 1.
I20260810 18:09:08.324055 6370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f5622f1452472a9c43988f6f782263" candidate_uuid: "70c4da52e8d044108d1aff802619c4d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e55952bf73b34a8ea5c0409283de7ab7"
I20260810 18:09:08.324219 6370 raft_consensus.cc:3037] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:08.324333 6005 leader_election.cc:302] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cecfd1fc63434dbb19d83c1d1ce5ac, 70c4da52e8d044108d1aff802619c4d5; no voters:
I20260810 18:09:08.324581 6135 raft_consensus.cc:2781] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:09:08.324786 6135 raft_consensus.cc:686] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 LEADER]: Becoming Leader. State: Replica: 70c4da52e8d044108d1aff802619c4d5, State: Running, Role: LEADER
I20260810 18:09:08.325109 6135 consensus_queue.cc:237] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:08.325652 6370 raft_consensus.cc:2454] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70c4da52e8d044108d1aff802619c4d5 in term 1.
I20260810 18:09:08.328334 5916 catalog_manager.cc:5495] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70c4da52e8d044108d1aff802619c4d5 (127.21.182.65). New cstate: current_term: 1 leader_uuid: "70c4da52e8d044108d1aff802619c4d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } health_report { overall_health: HEALTHY } } }
I20260810 18:09:08.357369 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.353719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.21.182.65:45561)
0810 18:09:08.353974 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:08.353978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:08.353992 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:09:08.354073 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:08.354075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:08.354128 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:08.354232 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:09:08.354472 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.355002 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.355821 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:08.356101 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:08.356183 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:08.356540 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:09:08.357004 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:09:08.357094 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:08.357159 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 18:09:08.357210 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":100,"threads_started":1}
I20260810 18:09:08.361722 6370 raft_consensus.cc:1261] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7 [term 1 FOLLOWER]: Refusing update from remote peer 70c4da52e8d044108d1aff802619c4d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:09:08.361763 6229 raft_consensus.cc:1261] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 1 FOLLOWER]: Refusing update from remote peer 70c4da52e8d044108d1aff802619c4d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:09:08.362550 6135 consensus_queue.cc:1035] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:09:08.362783 6450 consensus_queue.cc:1035] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:09:08.371624 6456 mvcc.cc:205] Tried to move back new op lower bound from 7317034386882232320 to 7317034386743136256. Current Snapshot: MvccSnapshot[applied={T|T < 7317034386882232320}]
I20260810 18:09:08.372601 6455 mvcc.cc:205] Tried to move back new op lower bound from 7317034386882232320 to 7317034386743136256. Current Snapshot: MvccSnapshot[applied={T|T < 7317034386882232320}]
I20260810 18:09:08.379562 5915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:09:08.379729 5915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:09:08.382274 5915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Sending DeleteTablet for 3 replicas of tablet 27f5622f1452472a9c43988f6f782263
I20260810 18:09:08.383052 6054 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f5622f1452472a9c43988f6f782263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:09:08 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:35780
I20260810 18:09:08.383054 6205 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f5622f1452472a9c43988f6f782263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:09:08 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:44196
I20260810 18:09:08.383472 6464 tablet_replica.cc:330] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5: stopping tablet replica
I20260810 18:09:08.383529 6349 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f5622f1452472a9c43988f6f782263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:09:08 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:60484
I20260810 18:09:08.383596 6465 tablet_replica.cc:330] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac: stopping tablet replica
I20260810 18:09:08.383742 6464 raft_consensus.cc:2229] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:09:08.384004 6465 raft_consensus.cc:2229] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:09:08.384239 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 5962
I20260810 18:09:08.388052 6467 tablet_replica.cc:330] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7: stopping tablet replica
I20260810 18:09:08.388428 6467 raft_consensus.cc:2229] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:09:08.388793 6467 raft_consensus.cc:2258] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:09:08.390311 6467 ts_tablet_manager.cc:1905] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:09:08.392055 6465 raft_consensus.cc:2258] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:09:08.393165 6467 ts_tablet_manager.cc:1918] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:09:08.393311 6467 log.cc:1196] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/27f5622f1452472a9c43988f6f782263
I20260810 18:09:08.393687 6467 ts_tablet_manager.cc:1939] T 27f5622f1452472a9c43988f6f782263 P e55952bf73b34a8ea5c0409283de7ab7: Deleting consensus metadata
I20260810 18:09:08.394044 6465 ts_tablet_manager.cc:1905] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:09:08.395241 5901 catalog_manager.cc:4841] TS e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773): tablet 27f5622f1452472a9c43988f6f782263 (table test-table [id=0446a97265204ff297f11af44e8a8cf5]) successfully deleted
W20260810 18:09:08.395943 5901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:45561 (error 108)
I20260810 18:09:08.396257 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 6133
I20260810 18:09:08.397897 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.397633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:45561 (local address 127.0.0.1:35800)
0810 18:09:08.397745 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:08.397818 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:45561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260810 18:09:08.398052 5901 catalog_manager.cc:4584] TS 70c4da52e8d044108d1aff802619c4d5 (127.21.182.65:45561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27f5622f1452472a9c43988f6f782263: Network error: Client connection negotiation failed: client connection to 127.21.182.65:45561: connect: Connection refused (error 111)
I20260810 18:09:08.406106 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 6281
I20260810 18:09:08.406664 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.406272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:34303 (local address 127.0.0.1:44202)
0810 18:09:08.406491 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:08.406556 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.66:34303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260810 18:09:08.417158 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 5883
I20260810 18:09:08.429620 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:44475
--webserver_interface=127.21.182.126
--webserver_port=38949
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:44475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:09:08.555935 6469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:08.556268 6469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:08.556314 6469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:08.560669 6469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:09:08.560756 6469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:08.560776 6469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:08.560794 6469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:09:08.560810 6469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:09:08.566639 6469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:44475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:44475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=38949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:08.568063 6469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:08.569298 6469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:08.578923 6474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:08.579035 6469 server_base.cc:1034] running on GCE node
W20260810 18:09:08.578958 6475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:08.578898 6479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:08.579941 6469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:08.581188 6469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:08.582399 6469 hybrid_clock.cc:648] HybridClock initialized: now 1786385348582367 us; error 50 us; skew 500 ppm
Aug 10 18:09:08 dist-test-slave-27sr krb5kdc[5246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385348, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:08.585554 6469 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:09:08.586644 6469 webserver.cc:460] Webserver started at http://127.21.182.126:38949/ using document root <none> and password file <none>
I20260810 18:09:08.587335 6469 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:08.587435 6469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:08.591826 6469 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:08.594521 6488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:08.596244 6469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:08.596405 6469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "3e8fd4fcef724555a349913e33c10286"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "0d7047b7bd5f6f686bb56aeb696da719"
server_key_iv: "9241dc5ce1bd13fd817d0042be058a0c"
server_key_version: "encryptionkey@0"
I20260810 18:09:08.597038 6469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:08.609761 6469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:08.613277 6469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:08.613523 6469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:08.621781 6469 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:44475
I20260810 18:09:08.622095 6550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:44475 every 8 connection(s)
I20260810 18:09:08.623091 6469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:09:08.627820 6551 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:09:08.628118 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 6469
I20260810 18:09:08.628604 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:45561
--local_ip_for_outbound_sockets=127.21.182.65
--tserver_master_addrs=127.21.182.126:44475
--webserver_port=38689
--webserver_interface=127.21.182.65
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:09:08.629807 6551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Bootstrap starting.
I20260810 18:09:08.648332 6551 log.cc:824] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:08.660543 6551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:09:08.661002 6551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: Bootstrap complete.
I20260810 18:09:08.665414 6551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } }
I20260810 18:09:08.665858 6551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8fd4fcef724555a349913e33c10286, State: Initialized, Role: FOLLOWER
I20260810 18:09:08.666440 6551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } }
I20260810 18:09:08.666592 6551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:09:08.666678 6551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:09:08.666798 6551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:09:08.668962 6551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } }
I20260810 18:09:08.669329 6551 leader_election.cc:302] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e8fd4fcef724555a349913e33c10286; no voters:
I20260810 18:09:08.669682 6551 leader_election.cc:288] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 18:09:08.669852 6556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:09:08.670125 6556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [term 3 LEADER]: Becoming Leader. State: Replica: 3e8fd4fcef724555a349913e33c10286, State: Running, Role: LEADER
I20260810 18:09:08.670499 6556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } }
I20260810 18:09:08.671056 6551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:09:08.671624 6556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e8fd4fcef724555a349913e33c10286. Latest consensus state: current_term: 3 leader_uuid: "3e8fd4fcef724555a349913e33c10286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } } }
I20260810 18:09:08.671774 6556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:08.672256 6557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3e8fd4fcef724555a349913e33c10286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8fd4fcef724555a349913e33c10286" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 44475 } } }
I20260810 18:09:08.672439 6557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286 [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:08.677057 6572 catalog_manager.cc:1259] Loaded cluster ID: e063ee4ac5624bb48ccc37b3601d0a06
I20260810 18:09:08.677138 6572 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: loading cluster ID for follower catalog manager: success
W20260810 18:09:08.678380 6572 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 3e8fd4fcef724555a349913e33c10286: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260810 18:09:08.678476 6572 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260810 18:09:08.678570 6561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:09:08.681640 6561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:09:08.682243 6561 catalog_manager.cc:1259] Loaded cluster ID: e063ee4ac5624bb48ccc37b3601d0a06
I20260810 18:09:08.682313 6561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 18:09:08.683089 6561 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdb046099ad google::LogMessage::Fail() at ??:0
@ 0x7fdb0460c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fdb046094de google::LogMessage::Flush() at ??:0
@ 0x7fdb0460aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdb0ef4cfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fdb0ef4dd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdb0ef4a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fdb0ef8e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fdb0685b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdb0685bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fdb0685d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fdb0684c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdb07e906db start_thread at ??:0
@ 0x7fdb0310761f clone at ??:0
W20260810 18:09:08.785152 6554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:08.785457 6554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:08.785503 6554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:08.789798 6554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:08.789882 6554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:08.789945 6554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:09:08.796036 6554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:45561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=38689
--tserver_master_addrs=127.21.182.126:44475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:08.797330 6554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:08.798472 6554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:08.809297 6596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:08.809297 6595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:08.809533 6600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:08.809602 6554 server_base.cc:1034] running on GCE node
I20260810 18:09:08.810060 6554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:08.810817 6554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:08.811983 6554 hybrid_clock.cc:648] HybridClock initialized: now 1786385348811954 us; error 57 us; skew 500 ppm
Aug 10 18:09:08 dist-test-slave-27sr krb5kdc[5246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385348, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:08.815161 6554 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:09:08.816375 6554 webserver.cc:460] Webserver started at http://127.21.182.65:38689/ using document root <none> and password file <none>
I20260810 18:09:08.817003 6554 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:08.817070 6554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:08.821475 6554 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:09:08.824010 6608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:08.825515 6554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:08.825664 6554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "70c4da52e8d044108d1aff802619c4d5"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "fd1e782ae19e84525ea4f3e24c6b2c86"
server_key_iv: "9ae3a24352f62a47a50bc881cbed13b2"
server_key_version: "encryptionkey@0"
I20260810 18:09:08.826275 6554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:08.846372 6554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:08.850257 6554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:08.850543 6554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:08.851339 6554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:08.852676 6615 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 18:09:08.857466 6554 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 18:09:08.857561 6554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260810 18:09:08.857640 6554 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 18:09:08.859579 6554 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 18:09:08.859658 6554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 18:09:08.859982 6615 tablet_bootstrap.cc:492] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5: Bootstrap starting.
I20260810 18:09:08.871644 6554 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:45561
I20260810 18:09:08.871848 6736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:45561 every 8 connection(s)
I20260810 18:09:08.873121 6554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:09:08.877925 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 6554
I20260810 18:09:08.878490 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:34303
--local_ip_for_outbound_sockets=127.21.182.66
--tserver_master_addrs=127.21.182.126:44475
--webserver_port=34321
--webserver_interface=127.21.182.66
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:09:08.883090 6615 log.cc:824] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:08.891242 6615 tablet_bootstrap.cc:492] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:09:08.891705 6615 tablet_bootstrap.cc:492] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5: Bootstrap complete.
I20260810 18:09:08.893056 6615 ts_tablet_manager.cc:1397] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5: Time spent bootstrapping tablet: real 0.033s user 0.027s sys 0.004s
I20260810 18:09:08.895779 6615 raft_consensus.cc:348] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:08.896421 6615 raft_consensus.cc:729] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c4da52e8d044108d1aff802619c4d5, State: Initialized, Role: FOLLOWER
I20260810 18:09:08.896991 6615 consensus_queue.cc:260] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:08.898072 6615 ts_tablet_manager.cc:1428] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260810 18:09:08.914405 6739 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:09:08.875878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:57575)
0810 18:09:08.876659 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:08.876734 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:08.877909 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:08.913541 (+ 35632us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":193,"threads_started":1}
W20260810 18:09:08.915082 6737 heartbeater.cc:644] Failed to heartbeat to 127.21.182.126:44475 (0 consecutive failures): Network error: Failed to ping master at 127.21.182.126:44475: Client connection negotiation failed: client connection to 127.21.182.126:44475: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 18:09:08.915999 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.915657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:59639)
0810 18:09:08.915802 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:08.915876 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 18:09:08.916676 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:08.916452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:40503)
0810 18:09:08.916570 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:08.916625 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 18:09:08.916857 6737 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:09:09.014070 6740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:09.014400 6740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:09.014488 6740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:09.018725 6740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:09.018821 6740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:09.018913 6740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:09:09.024534 6740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:34303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=34321
--tserver_master_addrs=127.21.182.126:44475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:09.025712 6740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:09.026830 6740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:09.037374 6751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:09.037372 6750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:09.037437 6740 server_base.cc:1034] running on GCE node
W20260810 18:09:09.037374 6753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:09.038131 6740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:09.038918 6740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:09.040082 6740 hybrid_clock.cc:648] HybridClock initialized: now 1786385349040065 us; error 31 us; skew 500 ppm
Aug 10 18:09:09 dist-test-slave-27sr krb5kdc[5246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385349, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:09.043258 6740 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:09:09.044328 6740 webserver.cc:460] Webserver started at http://127.21.182.66:34321/ using document root <none> and password file <none>
I20260810 18:09:09.044940 6740 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:09.045012 6740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:09.049345 6740 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:09:09.052114 6762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:09.053546 6740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:09.053704 6740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac"
format_stamp: "Formatted at 2026-08-10 18:09:06 on dist-test-slave-27sr"
server_key: "26839255bc1c4be8ab44132c335dd72d"
server_key_iv: "ecb26a6e7cfeba55d6123c3c7d8c6742"
server_key_version: "encryptionkey@0"
I20260810 18:09:09.054313 6740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:09.086270 6740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:09.090004 6740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:09.090283 6740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:09.091041 6740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:09.092376 6770 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260810 18:09:09.096894 6770 ts_tablet_manager.cc:1852] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260810 18:09:09.097357 6770 ts_tablet_manager.cc:1905] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:09:09.101766 6770 ts_tablet_manager.cc:1918] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:09:09.101897 6770 log.cc:1196] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/27f5622f1452472a9c43988f6f782263
I20260810 18:09:09.102190 6770 ts_tablet_manager.cc:1939] T 27f5622f1452472a9c43988f6f782263 P 07cecfd1fc63434dbb19d83c1d1ce5ac: Deleting consensus metadata
I20260810 18:09:09.102521 6740 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260810 18:09:09.102620 6740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260810 18:09:09.102725 6740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:09:09.102783 6740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:09.113202 6740 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:34303
I20260810 18:09:09.113240 6893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:34303 every 8 connection(s)
I20260810 18:09:09.114343 6740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:09:09.118247 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:09.116879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:48701)
0810 18:09:09.117405 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:09.117523 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
W20260810 18:09:09.118829 6895 heartbeater.cc:644] Failed to heartbeat to 127.21.182.126:44475 (0 consecutive failures): Network error: Failed to ping master at 127.21.182.126:44475: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
I20260810 18:09:09.119540 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:09.119289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:33763)
0810 18:09:09.119416 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:09.119480 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 18:09:09.120267 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:09.120079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:54571)
0810 18:09:09.120176 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:09.120219 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 18:09:09.120471 6895 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 18:09:09.121361 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 6740
I20260810 18:09:09.121785 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:34773
--local_ip_for_outbound_sockets=127.21.182.67
--tserver_master_addrs=127.21.182.126:44475
--webserver_port=35299
--webserver_interface=127.21.182.67
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:09:09.247828 6898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:09.248237 6898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:09.248322 6898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:09.252836 6898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:09.252923 6898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:09.253022 6898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:09:09.258930 6898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:34773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=35299
--tserver_master_addrs=127.21.182.126:44475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:09.260375 6898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:09.261559 6898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:09.272687 6907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:09.272706 6904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:09.272689 6903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:09.272806 6898 server_base.cc:1034] running on GCE node
I20260810 18:09:09.273443 6898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:09.274230 6898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:09.275413 6898 hybrid_clock.cc:648] HybridClock initialized: now 1786385349275394 us; error 34 us; skew 500 ppm
Aug 10 18:09:09 dist-test-slave-27sr krb5kdc[5246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385349, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:09.278700 6898 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:09:09.279721 6898 webserver.cc:460] Webserver started at http://127.21.182.67:35299/ using document root <none> and password file <none>
I20260810 18:09:09.280434 6898 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:09.280506 6898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:09.284796 6898 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:09:09.287278 6915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:09.288825 6898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:09:09.288995 6898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "e55952bf73b34a8ea5c0409283de7ab7"
format_stamp: "Formatted at 2026-08-10 18:09:07 on dist-test-slave-27sr"
server_key: "aac1d8e4bcb25fab835ccc0f4088c6c5"
server_key_iv: "b87129078146cd5a2cf9b0f328281280"
server_key_version: "encryptionkey@0"
I20260810 18:09:09.289594 6898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:09.310240 6898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:09.313854 6898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:09.314109 6898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:09.314862 6898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:09.315877 6898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:09:09.316012 6898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:09.316083 6898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:09:09.316118 6898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:09.326121 6898 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:34773
I20260810 18:09:09.326140 7044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:34773 every 8 connection(s)
I20260810 18:09:09.327306 6898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:09:09.331045 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 6898
I20260810 18:09:09.332190 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:09.329815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:50515)
0810 18:09:09.330363 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:09.330494 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":116,"threads_started":1}
W20260810 18:09:09.332710 7045 heartbeater.cc:644] Failed to heartbeat to 127.21.182.126:44475 (0 consecutive failures): Network error: Failed to ping master at 127.21.182.126:44475: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
I20260810 18:09:09.333400 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:09.333160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:60793)
0810 18:09:09.333277 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:09.333342 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 18:09:09.334059 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:09.333848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:55809)
0810 18:09:09.333957 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:09.334001 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 18:09:09.334226 7045 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 18:09:09.918087 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:09.917584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:33975)
0810 18:09:09.917878 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:09.917966 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
I20260810 18:09:10.121694 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:10.121194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:56427)
0810 18:09:10.121520 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:10.121585 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":100,"threads_started":1}
I20260810 18:09:10.154522 7051 raft_consensus.cc:482] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:09:10.154806 7051 raft_consensus.cc:504] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:10.156193 7051 leader_election.cc:288] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303), e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773)
Aug 10 18:09:10 dist-test-slave-27sr krb5kdc[5246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385348, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.67@KRBTEST.COM
Aug 10 18:09:10 dist-test-slave-27sr krb5kdc[5246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385348, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.66@KRBTEST.COM
I20260810 18:09:10.170643 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:10.156390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34773 (local address 127.21.182.65:60443)
0810 18:09:10.156997 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:10.157010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:10.157146 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:10.158804 (+ 1658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:10.158810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:10.159398 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:10.159832 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:10.159848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:10.161210 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:10.161214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:10.162127 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:10.162137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:10.162861 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:10.162864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:10.163028 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:10.163818 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:10.163831 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:10.165478 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:10.168090 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:10.168098 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:10.168113 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:10.168420 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:10.168796 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:10.168802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:10.168805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:10.168947 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:10.169633 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:10.169640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:10.169876 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 18:09:10.170066 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 18:09:10.170428 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":94,"thread_start_us":81,"threads_started":1}
I20260810 18:09:10.170657 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:10.156704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:60443 (local address 127.21.182.67:34773)
0810 18:09:10.156969 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:10.156988 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:10.157133 (+ 145us) server_negotiation.cc:407] Connection header received
0810 18:09:10.157955 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:10.157962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:10.158420 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:10.158745 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:10.160033 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:10.161046 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:10.162250 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:10.162725 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:10.162910 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:10.165733 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:10.165738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:10.165751 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:10.165779 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:10.167940 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:10.168556 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:10.168563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:10.168569 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:10.168696 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:10.169121 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:10.169123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:10.169124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:10.169504 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:10.169646 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:10.170195 (+ 549us) server_negotiation.cc:299] Negotiation successful
0810 18:09:10.170459 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
I20260810 18:09:10.170699 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:10.156769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:38461 (local address 127.21.182.66:34303)
0810 18:09:10.157235 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:10.157247 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:10.157283 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:09:10.158054 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:10.158061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:10.158443 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:10.158758 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:10.160068 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:10.161105 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:10.162312 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:10.162773 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:10.162950 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:10.166108 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:10.166113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:10.166125 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:10.166153 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:10.168152 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:10.168631 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:10.168643 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:10.168649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:10.168764 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:10.169072 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:10.169075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:10.169077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:10.169527 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:10.169676 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:10.170199 (+ 523us) server_negotiation.cc:299] Negotiation successful
0810 18:09:10.170498 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":88,"threads_started":1}
I20260810 18:09:10.170691 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:10.156385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:34303 (local address 127.21.182.65:38461)
0810 18:09:10.156612 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:10.156628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:10.156744 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:10.158803 (+ 2059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:10.158810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:10.159391 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:10.159833 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:10.159848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:10.161273 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:10.161317 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:10.162179 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:10.162189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:10.162914 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:10.162918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:10.163031 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:10.163771 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:10.163788 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:10.165901 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:10.168272 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:10.168275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:10.168278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:10.168505 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:10.168870 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:10.168873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:10.168875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:10.168972 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:10.169646 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:10.169649 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:10.169875 (+ 226us) client_negotiation.cc:769] Sending connection context
0810 18:09:10.170067 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 18:09:10.170516 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":284,"spinlock_wait_cycles":61440}
W20260810 18:09:10.173506 6612 leader_election.cc:341] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303): Not found: Tablet not found: 27f5622f1452472a9c43988f6f782263
W20260810 18:09:10.173780 6610 leader_election.cc:341] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773): Not found: Tablet not found: 27f5622f1452472a9c43988f6f782263
I20260810 18:09:10.173913 6610 leader_election.cc:302] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70c4da52e8d044108d1aff802619c4d5; no voters: 07cecfd1fc63434dbb19d83c1d1ce5ac, e55952bf73b34a8ea5c0409283de7ab7
I20260810 18:09:10.174201 7051 raft_consensus.cc:2726] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:09:10.335327 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:10.334799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:40031)
0810 18:09:10.335153 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:10.335213 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":125,"threads_started":1}
I20260810 18:09:10.919430 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:10.918971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:38185)
0810 18:09:10.919249 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:10.919308 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260810 18:09:11.122916 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:11.122494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:39081)
0810 18:09:11.122765 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:11.122822 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":122,"threads_started":1}
I20260810 18:09:11.336755 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:11.336288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:50089)
0810 18:09:11.336567 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:11.336631 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
I20260810 18:09:11.921066 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:11.920520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:41509)
0810 18:09:11.920893 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:11.920964 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":123,"threads_started":1}
I20260810 18:09:12.124413 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:12.123835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:46243)
0810 18:09:12.124220 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:12.124285 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":186,"threads_started":1}
I20260810 18:09:12.217640 7064 raft_consensus.cc:482] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:09:12.217825 7064 raft_consensus.cc:504] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:12.218251 7064 leader_election.cc:288] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303), e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773)
W20260810 18:09:12.219215 6612 leader_election.cc:341] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303): Not found: Tablet not found: 27f5622f1452472a9c43988f6f782263
W20260810 18:09:12.219363 6610 leader_election.cc:341] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773): Not found: Tablet not found: 27f5622f1452472a9c43988f6f782263
I20260810 18:09:12.219462 6610 leader_election.cc:302] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70c4da52e8d044108d1aff802619c4d5; no voters: 07cecfd1fc63434dbb19d83c1d1ce5ac, e55952bf73b34a8ea5c0409283de7ab7
I20260810 18:09:12.219640 7064 raft_consensus.cc:2726] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:09:12.338317 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:12.337780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:53703)
0810 18:09:12.338138 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:12.338209 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":91,"threads_started":1}
I20260810 18:09:12.922755 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:12.922195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:48801)
0810 18:09:12.922554 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:12.922631 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":94,"threads_started":1}
I20260810 18:09:13.125876 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:13.125382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:53429)
0810 18:09:13.125702 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:13.125771 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":111,"threads_started":1}
I20260810 18:09:13.339972 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:13.339345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:43633)
0810 18:09:13.339663 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:13.339792 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":89,"threads_started":1}
I20260810 18:09:13.924551 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:13.924012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:47987)
0810 18:09:13.924351 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:13.924451 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":78,"threads_started":1}
I20260810 18:09:14.127312 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:14.126820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:48119)
0810 18:09:14.127121 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:14.127198 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260810 18:09:14.341346 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:14.340908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:56797)
0810 18:09:14.341178 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:14.341242 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260810 18:09:14.502359 7073 raft_consensus.cc:482] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:09:14.502501 7073 raft_consensus.cc:504] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:14.502920 7073 leader_election.cc:288] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303), e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773)
W20260810 18:09:14.503976 6610 leader_election.cc:341] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773): Not found: Tablet not found: 27f5622f1452472a9c43988f6f782263
W20260810 18:09:14.504122 6612 leader_election.cc:341] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303): Not found: Tablet not found: 27f5622f1452472a9c43988f6f782263
I20260810 18:09:14.504201 6612 leader_election.cc:302] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70c4da52e8d044108d1aff802619c4d5; no voters: 07cecfd1fc63434dbb19d83c1d1ce5ac, e55952bf73b34a8ea5c0409283de7ab7
I20260810 18:09:14.504361 7073 raft_consensus.cc:2726] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:09:14.925940 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:14.925429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:55563)
0810 18:09:14.925750 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:14.925824 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
I20260810 18:09:15.128814 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:15.128338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:51643)
0810 18:09:15.128644 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:15.128712 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":113,"threads_started":1}
I20260810 18:09:15.342595 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:15.342169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:51759)
0810 18:09:15.342442 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:15.342499 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260810 18:09:15.927318 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:15.926864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:44677)
0810 18:09:15.927161 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:15.927217 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":97,"threads_started":1}
I20260810 18:09:16.130167 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:16.129601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:44533)
0810 18:09:16.129939 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:16.130005 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":81,"threads_started":1}
I20260810 18:09:16.344040 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:16.343441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:60251)
0810 18:09:16.343812 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:16.343882 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":100,"threads_started":1}
I20260810 18:09:16.928985 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:16.928461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:59803)
0810 18:09:16.928808 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:16.928879 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":103,"threads_started":1}
I20260810 18:09:17.131515 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:17.131107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:48381)
0810 18:09:17.131360 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:17.131417 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260810 18:09:17.345435 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:17.344913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:48403)
0810 18:09:17.345259 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:17.345327 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
I20260810 18:09:17.500424 7083 raft_consensus.cc:482] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:09:17.500594 7083 raft_consensus.cc:504] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:17.501077 7083 leader_election.cc:288] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303), e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773)
W20260810 18:09:17.501952 6612 leader_election.cc:341] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303): Not found: Tablet not found: 27f5622f1452472a9c43988f6f782263
W20260810 18:09:17.502063 6610 leader_election.cc:341] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773): Not found: Tablet not found: 27f5622f1452472a9c43988f6f782263
I20260810 18:09:17.502192 6610 leader_election.cc:302] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70c4da52e8d044108d1aff802619c4d5; no voters: 07cecfd1fc63434dbb19d83c1d1ce5ac, e55952bf73b34a8ea5c0409283de7ab7
I20260810 18:09:17.502372 7083 raft_consensus.cc:2726] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:09:17.930444 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:17.929954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:34039)
0810 18:09:17.930262 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:17.930336 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
I20260810 18:09:18.132860 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:18.132344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:43409)
0810 18:09:18.132642 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:18.132720 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260810 18:09:18.346829 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:18.346314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:51779)
0810 18:09:18.346637 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:18.346711 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
I20260810 18:09:18.932125 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:18.931497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:34575)
0810 18:09:18.931849 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:18.931971 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":114,"threads_started":1}
I20260810 18:09:19.134292 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:19.133836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:44151)
0810 18:09:19.134114 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:19.134182 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20260810 18:09:19.348409 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:19.347834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:60061)
0810 18:09:19.348215 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:19.348288 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":140,"threads_started":1}
I20260810 18:09:19.933490 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:19.933040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:35319)
0810 18:09:19.933323 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:19.933387 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
I20260810 18:09:20.135546 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:20.135104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:47465)
0810 18:09:20.135382 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:20.135444 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":95,"threads_started":1}
I20260810 18:09:20.349963 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:20.349383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:59593)
0810 18:09:20.349730 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:20.349811 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":104,"threads_started":1}
I20260810 18:09:20.935045 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:20.934440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:57279)
0810 18:09:20.934798 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:20.934890 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":88,"threads_started":1}
I20260810 18:09:20.960805 7099 raft_consensus.cc:482] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:09:20.960992 7099 raft_consensus.cc:504] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cecfd1fc63434dbb19d83c1d1ce5ac" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 34303 } } peers { permanent_uuid: "e55952bf73b34a8ea5c0409283de7ab7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34773 } } peers { permanent_uuid: "70c4da52e8d044108d1aff802619c4d5" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 45561 } }
I20260810 18:09:20.961499 7099 leader_election.cc:288] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303), e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773)
W20260810 18:09:20.962323 6612 leader_election.cc:341] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 07cecfd1fc63434dbb19d83c1d1ce5ac (127.21.182.66:34303): Not found: Tablet not found: 27f5622f1452472a9c43988f6f782263
W20260810 18:09:20.962482 6610 leader_election.cc:341] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e55952bf73b34a8ea5c0409283de7ab7 (127.21.182.67:34773): Not found: Tablet not found: 27f5622f1452472a9c43988f6f782263
I20260810 18:09:20.962561 6610 leader_election.cc:302] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70c4da52e8d044108d1aff802619c4d5; no voters: 07cecfd1fc63434dbb19d83c1d1ce5ac, e55952bf73b34a8ea5c0409283de7ab7
I20260810 18:09:20.962702 7099 raft_consensus.cc:2726] T 27f5622f1452472a9c43988f6f782263 P 70c4da52e8d044108d1aff802619c4d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:09:21.137730 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:21.136667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:52079)
0810 18:09:21.137528 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:21.137622 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":652,"threads_started":1}
I20260810 18:09:21.351583 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:21.351033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:59387)
0810 18:09:21.351369 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:21.351475 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":89,"threads_started":1}
I20260810 18:09:21.936632 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:21.936143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:49133)
0810 18:09:21.936462 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:21.936528 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":93,"threads_started":1}
I20260810 18:09:22.139235 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:22.138761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:57851)
0810 18:09:22.139069 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:22.139131 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":101,"threads_started":1}
I20260810 18:09:22.352941 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:22.352451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:58849)
0810 18:09:22.352737 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:22.352803 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
I20260810 18:09:22.938097 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:22.937525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:41987)
0810 18:09:22.937874 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:22.937965 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":123,"threads_started":1}
I20260810 18:09:23.140743 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:23.140232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:41577)
0810 18:09:23.140564 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:23.140635 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260810 18:09:23.354420 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:23.353904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:53801)
0810 18:09:23.354233 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:23.354302 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":113,"threads_started":1}
I20260810 18:09:23.939778 7109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:23.939311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.65:33751)
0810 18:09:23.939612 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:23.939676 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":103,"threads_started":1}
I20260810 18:09:24.142149 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:24.141715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.66:38847)
0810 18:09:24.141983 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:24.142050 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260810 18:09:24.340186 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 6554
I20260810 18:09:24.351965 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 6740
I20260810 18:09:24.356063 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:24.355524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:44475 (local address 127.21.182.67:46607)
0810 18:09:24.355835 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:24.355949 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
I20260810 18:09:24.361370 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 6898
2026-08-10T18:09:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19606 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:09:24 dist-test-slave-27sr krb5kdc[7114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:09:24 dist-test-slave-27sr krb5kdc[7114](info): set up 2 sockets
Aug 10 18:09:24 dist-test-slave-27sr krb5kdc[7114](info): commencing operation
krb5kdc: starting...
W20260810 18:09:25.563270 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.003s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:09:25 dist-test-slave-27sr krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:09:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:09:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:09:25.706842 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:33733
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:09:25.828068 7132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:25.828401 7132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:25.828446 7132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:25.832895 7132 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 18:09:25.832985 7132 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 18:09:25.833002 7132 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 18:09:25.833016 7132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:09:25.833037 7132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:25.833056 7132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:25.833122 7132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:09:25.833144 7132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:09:25.839200 7132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:33733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:25.840591 7132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:25.841820 7132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:25.851975 7141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:25.851964 7137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:25.852025 7132 server_base.cc:1034] running on GCE node
W20260810 18:09:25.851964 7138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:25.853116 7132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:25.854476 7132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:25.855685 7132 hybrid_clock.cc:648] HybridClock initialized: now 1786385365855662 us; error 41 us; skew 500 ppm
Aug 10 18:09:25 dist-test-slave-27sr krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385365, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:25.858975 7132 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:09:25.860059 7132 webserver.cc:460] Webserver started at http://127.21.182.126:33425/ using document root <none> and password file <none>
I20260810 18:09:25.860726 7132 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:25.860836 7132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:25.861089 7132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:09:25.863337 7132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "0967adc98d744d208e5b223ac1e5bfea"
format_stamp: "Formatted at 2026-08-10 18:09:25 on dist-test-slave-27sr"
server_key: "c6e7f64b79bf022671bc8ca1238d22ee"
server_key_iv: "cfa2a24bcab1c6a86dcfc654f9f4ab82"
server_key_version: "encryptionkey@0"
I20260810 18:09:25.864038 7132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "0967adc98d744d208e5b223ac1e5bfea"
format_stamp: "Formatted at 2026-08-10 18:09:25 on dist-test-slave-27sr"
server_key: "c6e7f64b79bf022671bc8ca1238d22ee"
server_key_iv: "cfa2a24bcab1c6a86dcfc654f9f4ab82"
server_key_version: "encryptionkey@0"
I20260810 18:09:25.868304 7132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:09:25.871182 7148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:25.872653 7132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:25.872812 7132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "0967adc98d744d208e5b223ac1e5bfea"
format_stamp: "Formatted at 2026-08-10 18:09:25 on dist-test-slave-27sr"
server_key: "c6e7f64b79bf022671bc8ca1238d22ee"
server_key_iv: "cfa2a24bcab1c6a86dcfc654f9f4ab82"
server_key_version: "encryptionkey@0"
I20260810 18:09:25.872957 7132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:25.895603 7132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:25.899406 7132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:25.899655 7132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:25.907960 7132 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:33733
I20260810 18:09:25.908378 7204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:33733 every 8 connection(s)
I20260810 18:09:25.909088 7132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:09:25.912506 7207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:25.915395 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 7132
I20260810 18:09:25.915557 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:09:25.915881 22233 external_mini_cluster.cc:1428] Setting key eccddc615395280c5b96a68b09a708c4
I20260810 18:09:25.919768 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Bootstrap starting.
I20260810 18:09:25.922492 7207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:25.923384 7207 log.cc:824] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:09:25 dist-test-slave-27sr krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:25.926160 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: No bootstrap required, opened a new log
I20260810 18:09:25.930877 7207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } }
I20260810 18:09:25.931221 7207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:25.931326 7207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0967adc98d744d208e5b223ac1e5bfea, State: Initialized, Role: FOLLOWER
I20260810 18:09:25.931941 7207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } }
I20260810 18:09:25.932128 7207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:09:25.932132 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:25.917501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.21.182.126:33733)
0810 18:09:25.918016 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:25.918032 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:25.918072 (+ 40us) server_negotiation.cc:407] Connection header received
0810 18:09:25.918791 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:25.918811 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:25.919172 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:25.919545 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:25.920526 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:25.921576 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:25.922490 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:25.923051 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:25.923233 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:25.926270 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:25.926288 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:25.926303 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:25.926333 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:25.928261 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:25.928924 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:25.928929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:25.928935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:25.929065 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:25.929408 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:25.929411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:25.929412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:25.929962 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:25.930143 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:25.930534 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 18:09:25.931096 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":156,"threads_started":1}
I20260810 18:09:25.932253 7207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:09:25.932377 7207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:25.933686 7207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } }
I20260810 18:09:25.934089 7207 leader_election.cc:302] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0967adc98d744d208e5b223ac1e5bfea; no voters:
I20260810 18:09:25.934448 7207 leader_election.cc:288] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:09:25.934568 7212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:09:25.934904 7212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 1 LEADER]: Becoming Leader. State: Replica: 0967adc98d744d208e5b223ac1e5bfea, State: Running, Role: LEADER
I20260810 18:09:25.935211 7212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } }
I20260810 18:09:25.935947 7207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:09:25.936517 7213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0967adc98d744d208e5b223ac1e5bfea. Latest consensus state: current_term: 1 leader_uuid: "0967adc98d744d208e5b223ac1e5bfea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } } }
I20260810 18:09:25.936640 7213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:25.936949 7214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0967adc98d744d208e5b223ac1e5bfea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } } }
I20260810 18:09:25.937173 7216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:09:25.937178 7214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:25.940601 7216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:09:25.946372 7216 catalog_manager.cc:1347] Generated new cluster ID: e2465b5083454415ad78cdacbb8fc767
I20260810 18:09:25.946444 7216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:09:25.969127 7216 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:09:25.970644 7216 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:09:25.976877 7216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Generated new TSK 0
I20260810 18:09:25.977872 7216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:09:26.020603 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33733
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:09:26.144416 7236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:26.144767 7236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:26.144851 7236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:26.149066 7236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:26.149170 7236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:26.149276 7236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:09:26.155117 7236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:26.156522 7236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:26.157683 7236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:26.169339 7244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:26.169339 7241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:26.169399 7242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:26.169723 7236 server_base.cc:1034] running on GCE node
I20260810 18:09:26.170178 7236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:26.170848 7236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:26.171981 7236 hybrid_clock.cc:648] HybridClock initialized: now 1786385366171955 us; error 39 us; skew 500 ppm
Aug 10 18:09:26 dist-test-slave-27sr krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385366, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:26.175417 7236 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:09:26.176695 7236 webserver.cc:460] Webserver started at http://127.21.182.65:44319/ using document root <none> and password file <none>
I20260810 18:09:26.177346 7236 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:26.177413 7236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:26.177726 7236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:09:26.180526 7236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "71db6faa5f744b87bd63c27f00c3f20e"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "60aecb92391c73d1ffb7201d18281b6d"
server_key_iv: "bfde7873ad98854cc60ba5362f985ad9"
server_key_version: "encryptionkey@0"
I20260810 18:09:26.181221 7236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "71db6faa5f744b87bd63c27f00c3f20e"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "60aecb92391c73d1ffb7201d18281b6d"
server_key_iv: "bfde7873ad98854cc60ba5362f985ad9"
server_key_version: "encryptionkey@0"
I20260810 18:09:26.185645 7236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:09:26.188652 7253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:26.190297 7236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:26.190430 7236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "71db6faa5f744b87bd63c27f00c3f20e"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "60aecb92391c73d1ffb7201d18281b6d"
server_key_iv: "bfde7873ad98854cc60ba5362f985ad9"
server_key_version: "encryptionkey@0"
I20260810 18:09:26.190555 7236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:26.215716 7236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:26.219770 7236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:26.220036 7236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:26.220746 7236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:26.221719 7236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:09:26.221771 7236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:26.221817 7236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:09:26.221873 7236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:26.232208 7236 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:40959
I20260810 18:09:26.232252 7379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:40959 every 8 connection(s)
I20260810 18:09:26.233268 7236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:09:26.240257 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 7236
I20260810 18:09:26.240465 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:09:26.240796 22233 external_mini_cluster.cc:1428] Setting key 4a84e1b8133659fbd59d0a3732023147
Aug 10 18:09:26 dist-test-slave-27sr krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385366, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:26.252101 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.235486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:41723 (local address 127.21.182.126:33733)
0810 18:09:26.235694 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:26.235699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:26.236796 (+ 1097us) server_negotiation.cc:407] Connection header received
0810 18:09:26.238521 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:26.238524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:26.238584 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:26.238686 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:26.240318 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.241109 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.242748 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.243002 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.243096 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.247061 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:26.247065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:26.247067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:26.247097 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:26.249037 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:26.249869 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:26.249872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:26.249874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:26.249964 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:26.250507 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:26.250510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:26.250512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:26.250765 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:26.250865 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:26.251735 (+ 870us) server_negotiation.cc:299] Negotiation successful
0810 18:09:26.251884 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 18:09:26.253651 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.235965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:41723)
0810 18:09:26.236577 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:26.236665 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:26.238266 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:26.238960 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:26.238969 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:26.239411 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:26.240120 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.240145 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.241282 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.241285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.242600 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.242613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.243206 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.243214 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.243477 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.244337 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:26.244369 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:26.246837 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:26.249200 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:26.249212 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:26.249239 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:26.249719 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:26.250079 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:26.250086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:26.250091 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:26.250370 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:26.250906 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:26.250917 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:26.251381 (+ 464us) client_negotiation.cc:769] Sending connection context
0810 18:09:26.251652 (+ 271us) client_negotiation.cc:240] Negotiation successful
0810 18:09:26.252271 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
I20260810 18:09:26.255213 7380 heartbeater.cc:347] Connected to a master server at 127.21.182.126:33733
I20260810 18:09:26.255566 7380 heartbeater.cc:464] Registering TS with master...
I20260810 18:09:26.256305 7380 heartbeater.cc:511] Master 127.21.182.126:33733 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
I20260810 18:09:26.258594 7166 ts_manager.cc:194] Registered new tserver with Master: 71db6faa5f744b87bd63c27f00c3f20e (127.21.182.65:40959)
I20260810 18:09:26.260641 7166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:41723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:09:26.277664 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33733
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:09:26.400450 7385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:26.400743 7385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:26.400789 7385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:26.405031 7385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:26.405107 7385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:26.405170 7385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:09:26.410753 7385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:26.412055 7385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:26.413210 7385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:26.424671 7390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:26.424671 7391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:26.424671 7393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:26.424933 7385 server_base.cc:1034] running on GCE node
I20260810 18:09:26.425555 7385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:26.426307 7385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:26.427516 7385 hybrid_clock.cc:648] HybridClock initialized: now 1786385366427487 us; error 43 us; skew 500 ppm
Aug 10 18:09:26 dist-test-slave-27sr krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385366, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:26.430841 7385 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:09:26.432099 7385 webserver.cc:460] Webserver started at http://127.21.182.66:39183/ using document root <none> and password file <none>
I20260810 18:09:26.432776 7385 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:26.432883 7385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:26.433154 7385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:09:26.435559 7385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "3edfb4dd970b44ab919afc71b7cff75d"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "2ee9aaef8d0afa78ea8b917eca40d1bd"
server_key_iv: "b149a8508bf269dbb55a829653c401ef"
server_key_version: "encryptionkey@0"
I20260810 18:09:26.436547 7385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "3edfb4dd970b44ab919afc71b7cff75d"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "2ee9aaef8d0afa78ea8b917eca40d1bd"
server_key_iv: "b149a8508bf269dbb55a829653c401ef"
server_key_version: "encryptionkey@0"
I20260810 18:09:26.441610 7385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 18:09:26.444588 7402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:26.446161 7385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:09:26.446341 7385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "3edfb4dd970b44ab919afc71b7cff75d"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "2ee9aaef8d0afa78ea8b917eca40d1bd"
server_key_iv: "b149a8508bf269dbb55a829653c401ef"
server_key_version: "encryptionkey@0"
I20260810 18:09:26.446488 7385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:26.492650 7385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:26.497694 7385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:26.497936 7385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:26.498636 7385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:26.499624 7385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:09:26.499678 7385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:26.499752 7385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:09:26.499788 7385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:26.509477 7385 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:41385
I20260810 18:09:26.509506 7527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:41385 every 8 connection(s)
I20260810 18:09:26.510555 7385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:09:26.517350 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 7385
I20260810 18:09:26.517465 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:09:26.517760 22233 external_mini_cluster.cc:1428] Setting key 04c380c5a720d052c0a1bb54e06afb97
Aug 10 18:09:26 dist-test-slave-27sr krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385366, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:26.528479 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.513048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:58893 (local address 127.21.182.126:33733)
0810 18:09:26.513249 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:26.513253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:26.514207 (+ 954us) server_negotiation.cc:407] Connection header received
0810 18:09:26.516033 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:26.516036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:26.516106 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:26.516193 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:26.517699 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.518436 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.519712 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.520055 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.520175 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.523170 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:26.523177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:26.523183 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:26.523220 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:26.525406 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:26.526201 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:26.526208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:26.526212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:26.526332 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:26.526832 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:26.526839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:26.526844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:26.527162 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:26.527278 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:26.528083 (+ 805us) server_negotiation.cc:299] Negotiation successful
0810 18:09:26.528258 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 18:09:26.529474 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.513478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:58893)
0810 18:09:26.514008 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:26.514059 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:26.515526 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:26.516436 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:26.516444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:26.516880 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:26.517507 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.517530 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.518573 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.518577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.519571 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.519581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.520223 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.520228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.520419 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.521102 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:26.521121 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:26.522912 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:26.525585 (+ 2673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:26.525593 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:26.525610 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:26.526062 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:26.526506 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:26.526509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:26.526511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:26.526709 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:26.527298 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:26.527305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:26.527624 (+ 319us) client_negotiation.cc:769] Sending connection context
0810 18:09:26.528006 (+ 382us) client_negotiation.cc:240] Negotiation successful
0810 18:09:26.528371 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":134,"threads_started":1}
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
I20260810 18:09:26.531091 7528 heartbeater.cc:347] Connected to a master server at 127.21.182.126:33733
I20260810 18:09:26.531390 7528 heartbeater.cc:464] Registering TS with master...
I20260810 18:09:26.532127 7528 heartbeater.cc:511] Master 127.21.182.126:33733 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
I20260810 18:09:26.533638 7166 ts_manager.cc:194] Registered new tserver with Master: 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66:41385)
I20260810 18:09:26.534320 7166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:58893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:09:26.552071 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33733
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:09:26.674140 7533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:26.674476 7533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:26.674523 7533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:26.679062 7533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:26.679147 7533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:26.679243 7533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:09:26.685341 7533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:26.686585 7533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:26.687714 7533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:26.698580 7539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:26.698603 7541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:26.698593 7538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:26.698635 7533 server_base.cc:1034] running on GCE node
I20260810 18:09:26.699329 7533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:26.700109 7533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:26.701287 7533 hybrid_clock.cc:648] HybridClock initialized: now 1786385366701264 us; error 32 us; skew 500 ppm
Aug 10 18:09:26 dist-test-slave-27sr krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385366, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:26.704432 7533 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:09:26.705399 7533 webserver.cc:460] Webserver started at http://127.21.182.67:43847/ using document root <none> and password file <none>
I20260810 18:09:26.705974 7533 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:26.706036 7533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:26.706219 7533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:09:26.708536 7533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "878595c0cf284b1e919266439baf64aa"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "ecc9d69adddfe12974f6229c851bd424"
server_key_iv: "149061d30ae5ee5ff00740bcfcf3e901"
server_key_version: "encryptionkey@0"
I20260810 18:09:26.709111 7533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "878595c0cf284b1e919266439baf64aa"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "ecc9d69adddfe12974f6229c851bd424"
server_key_iv: "149061d30ae5ee5ff00740bcfcf3e901"
server_key_version: "encryptionkey@0"
I20260810 18:09:26.713326 7533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:09:26.716180 7550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:26.717670 7533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:26.717789 7533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "878595c0cf284b1e919266439baf64aa"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "ecc9d69adddfe12974f6229c851bd424"
server_key_iv: "149061d30ae5ee5ff00740bcfcf3e901"
server_key_version: "encryptionkey@0"
I20260810 18:09:26.717954 7533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:26.740728 7533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:26.744335 7533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:26.744567 7533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:26.745230 7533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:26.746160 7533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:09:26.746210 7533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:26.746287 7533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:09:26.746327 7533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:26.756678 7533 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:34131
I20260810 18:09:26.756716 7682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:34131 every 8 connection(s)
I20260810 18:09:26.757690 7533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:09:26.760896 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 7533
I20260810 18:09:26.761067 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:09:26.761396 22233 external_mini_cluster.cc:1428] Setting key c6e3fcb0f7f5cb035edc08b6af31fe0e
Aug 10 18:09:26 dist-test-slave-27sr krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385366, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:26.773100 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.759967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:38339 (local address 127.21.182.126:33733)
0810 18:09:26.760112 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:26.760116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:26.761103 (+ 987us) server_negotiation.cc:407] Connection header received
0810 18:09:26.762824 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:26.762828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:26.762892 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:26.762974 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:26.764486 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.765033 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.766218 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.766451 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.766569 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.769308 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:26.769311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:26.769313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:26.769340 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:26.770766 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:26.771366 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:26.771370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:26.771371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:26.771453 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:26.771829 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:26.771834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:26.771836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:26.772101 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:26.772214 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:26.772765 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 18:09:26.772918 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:09:26.773965 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.760359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:38339)
0810 18:09:26.760918 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:26.760963 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:26.762504 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:26.763165 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:26.763174 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:26.763705 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:26.764336 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.764350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.765171 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.765174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.766084 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.766093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.766613 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.766617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.766799 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.767450 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:26.767468 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:26.769117 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:26.770921 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:26.770928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:26.770948 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:26.771238 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:26.771552 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:26.771556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:26.771558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:26.771714 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:26.772214 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:26.772222 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:26.772508 (+ 286us) client_negotiation.cc:769] Sending connection context
0810 18:09:26.772709 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 18:09:26.773074 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":194,"threads_started":1}
I20260810 18:09:26.775388 7683 heartbeater.cc:347] Connected to a master server at 127.21.182.126:33733
I20260810 18:09:26.775676 7683 heartbeater.cc:464] Registering TS with master...
I20260810 18:09:26.776427 7683 heartbeater.cc:511] Master 127.21.182.126:33733 requested a full tablet report, sending...
I20260810 18:09:26.777499 7166 ts_manager.cc:194] Registered new tserver with Master: 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131)
I20260810 18:09:26.778052 7166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:38339
I20260810 18:09:26.788900 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:09:26.799285 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.791067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.21.182.126:33733)
0810 18:09:26.791231 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:26.791237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:26.791401 (+ 164us) server_negotiation.cc:407] Connection header received
0810 18:09:26.791573 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:26.791576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:26.791632 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:26.791714 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:26.792529 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.793064 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.793926 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.794265 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.794379 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.795934 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:26.795938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:26.795942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:26.795971 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:26.797427 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:26.797880 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:26.797885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:26.797887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:26.797983 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:26.798290 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:26.798295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:26.798297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:26.798550 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:26.798712 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:26.798979 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 18:09:26.799112 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:09:26.802786 7166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:09:26.806108 7166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:09:26.822540 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.815741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:41385 (local address 127.0.0.1:42874)
0810 18:09:26.816251 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:26.816317 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:26.816479 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:26.817432 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:26.817439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:26.817471 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:26.817657 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.817665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.819189 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.819194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.820655 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.820670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.821983 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.821987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.822065 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.822085 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:09:26.822166 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 18:09:26.822335 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":1,"thread_start_us":144,"threads_started":1}
I20260810 18:09:26.822587 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.816861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.21.182.67:34131)
0810 18:09:26.817685 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:26.817697 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:26.817720 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:09:26.817807 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:26.817812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:26.817953 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:26.818247 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:26.818418 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.819578 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.821088 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.821717 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.821834 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.821993 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:26.822252 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 18:09:26.822420 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":134,"threads_started":1}
I20260810 18:09:26.822541 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.816166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34131 (local address 127.0.0.1:48760)
0810 18:09:26.816955 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:26.816972 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:26.817064 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:26.818129 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:26.818131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:26.818144 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:26.818310 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.818322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.819758 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.819762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.820961 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.820971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.821870 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.821876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.822014 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.822065 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 18:09:26.822210 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 18:09:26.822368 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":120,"thread_start_us":131,"threads_started":1}
I20260810 18:09:26.823318 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.816606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:40959 (local address 127.0.0.1:47316)
0810 18:09:26.817316 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:26.817332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:26.817420 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:26.818041 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:26.818047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:26.818070 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:26.818220 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.818228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.819768 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.819772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.821489 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.821501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.822821 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.822824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.822937 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.822973 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 18:09:26.823055 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 18:09:26.823139 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":115,"thread_start_us":100,"threads_started":1}
I20260810 18:09:26.823676 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.815980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.21.182.66:41385)
0810 18:09:26.816547 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:26.816562 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:26.816590 (+ 28us) server_negotiation.cc:407] Connection header received
0810 18:09:26.816779 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:26.816943 (+ 164us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:26.817219 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:26.817851 (+ 632us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:26.818046 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.819036 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.820812 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.821847 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.823215 (+ 1368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.823368 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:26.823422 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 18:09:26.823516 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":179,"threads_started":1}
I20260810 18:09:26.823763 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.817001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.21.182.65:40959)
0810 18:09:26.817454 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:26.817461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:26.817483 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:09:26.817563 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:26.817568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:26.817879 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:26.818046 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:26.818376 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.819589 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.821632 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.822480 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.822590 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.822748 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:26.823511 (+ 763us) server_negotiation.cc:299] Negotiation successful
0810 18:09:26.823619 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":84,"threads_started":1}
I20260810 18:09:26.826553 7604 tablet_service.cc:1467] Processing CreateTablet for tablet 8d08370266b24a0e8691d56ff4b6ea5a (DEFAULT_TABLE table=test-table [id=e68a1dd8ee4a4365961f1d13094a0c2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:09:26.826848 7308 tablet_service.cc:1467] Processing CreateTablet for tablet 8d08370266b24a0e8691d56ff4b6ea5a (DEFAULT_TABLE table=test-table [id=e68a1dd8ee4a4365961f1d13094a0c2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:09:26.828017 7308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d08370266b24a0e8691d56ff4b6ea5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:26.828020 7604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d08370266b24a0e8691d56ff4b6ea5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:26.829572 7457 tablet_service.cc:1467] Processing CreateTablet for tablet 8d08370266b24a0e8691d56ff4b6ea5a (DEFAULT_TABLE table=test-table [id=e68a1dd8ee4a4365961f1d13094a0c2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:09:26.830606 7457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d08370266b24a0e8691d56ff4b6ea5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:26.836666 7702 tablet_bootstrap.cc:492] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: Bootstrap starting.
I20260810 18:09:26.837661 7703 tablet_bootstrap.cc:492] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa: Bootstrap starting.
I20260810 18:09:26.837940 7704 tablet_bootstrap.cc:492] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d: Bootstrap starting.
I20260810 18:09:26.840201 7703 tablet_bootstrap.cc:654] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:26.840214 7702 tablet_bootstrap.cc:654] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:26.840713 7704 tablet_bootstrap.cc:654] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:26.841218 7702 log.cc:824] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:26.841449 7703 log.cc:824] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:26.841523 7704 log.cc:824] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:26.843430 7704 tablet_bootstrap.cc:492] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d: No bootstrap required, opened a new log
I20260810 18:09:26.843500 7702 tablet_bootstrap.cc:492] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: No bootstrap required, opened a new log
I20260810 18:09:26.843500 7703 tablet_bootstrap.cc:492] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa: No bootstrap required, opened a new log
I20260810 18:09:26.843675 7704 ts_tablet_manager.cc:1397] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:09:26.843693 7702 ts_tablet_manager.cc:1397] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:09:26.843693 7703 ts_tablet_manager.cc:1397] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:09:26.847555 7704 raft_consensus.cc:348] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } }
I20260810 18:09:26.847882 7704 raft_consensus.cc:374] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:26.847965 7704 raft_consensus.cc:729] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3edfb4dd970b44ab919afc71b7cff75d, State: Initialized, Role: FOLLOWER
I20260810 18:09:26.847815 7703 raft_consensus.cc:348] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } }
I20260810 18:09:26.848136 7703 raft_consensus.cc:374] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:26.848208 7703 raft_consensus.cc:729] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878595c0cf284b1e919266439baf64aa, State: Initialized, Role: FOLLOWER
I20260810 18:09:26.848289 7702 raft_consensus.cc:348] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } }
I20260810 18:09:26.848536 7702 raft_consensus.cc:374] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:26.848594 7702 raft_consensus.cc:729] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71db6faa5f744b87bd63c27f00c3f20e, State: Initialized, Role: FOLLOWER
I20260810 18:09:26.848606 7704 consensus_queue.cc:260] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } }
I20260810 18:09:26.848680 7703 consensus_queue.cc:260] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } }
I20260810 18:09:26.849000 7702 consensus_queue.cc:260] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } }
I20260810 18:09:26.849505 7683 heartbeater.cc:503] Master 127.21.182.126:33733 was elected leader, sending a full tablet report...
I20260810 18:09:26.849844 7703 ts_tablet_manager.cc:1428] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:09:26.849844 7702 ts_tablet_manager.cc:1428] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260810 18:09:26.850035 7704 ts_tablet_manager.cc:1428] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:09:26.850193 7380 heartbeater.cc:503] Master 127.21.182.126:33733 was elected leader, sending a full tablet report...
I20260810 18:09:26.850397 7528 heartbeater.cc:503] Master 127.21.182.126:33733 was elected leader, sending a full tablet report...
I20260810 18:09:26.901254 7711 raft_consensus.cc:482] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:09:26.901520 7711 raft_consensus.cc:504] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } }
I20260810 18:09:26.902897 7711 leader_election.cc:288] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71db6faa5f744b87bd63c27f00c3f20e (127.21.182.65:40959), 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131)
I20260810 18:09:26.908341 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.903122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:40959 (local address 127.21.182.66:60287)
0810 18:09:26.903349 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:26.903370 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:26.903483 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:26.904244 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:26.904250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:26.904269 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:26.904363 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.904372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.905419 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.905424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.906838 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.906851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.907854 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.907858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.907995 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.908022 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:09:26.908108 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 18:09:26.908183 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":64}
I20260810 18:09:26.908357 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.903578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:60287 (local address 127.21.182.65:40959)
0810 18:09:26.903713 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:26.903716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:26.903729 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:09:26.903782 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:26.903785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:26.903826 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:26.903941 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:26.904484 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.905248 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.906984 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.907618 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.907751 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.907808 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:26.908147 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 18:09:26.908233 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:09:26.908393 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.903381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:37307 (local address 127.21.182.67:34131)
0810 18:09:26.903555 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:26.903560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:26.903797 (+ 237us) server_negotiation.cc:407] Connection header received
0810 18:09:26.903961 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:26.903972 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:26.904019 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:26.904116 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:26.904445 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.905248 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.906984 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.907618 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.907751 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.907808 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:26.908203 (+ 395us) server_negotiation.cc:299] Negotiation successful
0810 18:09:26.908277 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:09:26.908792 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.903204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34131 (local address 127.21.182.66:37307)
0810 18:09:26.903675 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:26.903687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:26.903829 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:26.904131 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:26.904134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:26.904154 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:26.904300 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.904308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.905424 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.905427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.906838 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:26.906851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.907942 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.907946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:26.908042 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.908061 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:09:26.908521 (+ 460us) client_negotiation.cc:240] Negotiation successful
0810 18:09:26.908616 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":71,"thread_start_us":147,"threads_started":1}
I20260810 18:09:26.909298 7328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d08370266b24a0e8691d56ff4b6ea5a" candidate_uuid: "3edfb4dd970b44ab919afc71b7cff75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71db6faa5f744b87bd63c27f00c3f20e" is_pre_election: true
I20260810 18:09:26.909571 7628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d08370266b24a0e8691d56ff4b6ea5a" candidate_uuid: "3edfb4dd970b44ab919afc71b7cff75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878595c0cf284b1e919266439baf64aa" is_pre_election: true
I20260810 18:09:26.909653 7328 raft_consensus.cc:2454] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3edfb4dd970b44ab919afc71b7cff75d in term 0.
I20260810 18:09:26.909852 7628 raft_consensus.cc:2454] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3edfb4dd970b44ab919afc71b7cff75d in term 0.
I20260810 18:09:26.910249 7405 leader_election.cc:302] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3edfb4dd970b44ab919afc71b7cff75d, 71db6faa5f744b87bd63c27f00c3f20e; no voters:
I20260810 18:09:26.910585 7711 raft_consensus.cc:2781] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:09:26.910676 7711 raft_consensus.cc:482] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:09:26.910728 7711 raft_consensus.cc:3037] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:26.911794 7711 raft_consensus.cc:504] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } }
I20260810 18:09:26.912228 7711 leader_election.cc:288] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [CANDIDATE]: Term 1 election: Requested vote from peers 71db6faa5f744b87bd63c27f00c3f20e (127.21.182.65:40959), 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131)
I20260810 18:09:26.912626 7628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d08370266b24a0e8691d56ff4b6ea5a" candidate_uuid: "3edfb4dd970b44ab919afc71b7cff75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878595c0cf284b1e919266439baf64aa"
I20260810 18:09:26.912624 7328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d08370266b24a0e8691d56ff4b6ea5a" candidate_uuid: "3edfb4dd970b44ab919afc71b7cff75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71db6faa5f744b87bd63c27f00c3f20e"
I20260810 18:09:26.912770 7628 raft_consensus.cc:3037] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:26.912772 7328 raft_consensus.cc:3037] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:26.914004 7328 raft_consensus.cc:2454] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3edfb4dd970b44ab919afc71b7cff75d in term 1.
I20260810 18:09:26.914081 7628 raft_consensus.cc:2454] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3edfb4dd970b44ab919afc71b7cff75d in term 1.
I20260810 18:09:26.914435 7403 leader_election.cc:302] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3edfb4dd970b44ab919afc71b7cff75d, 71db6faa5f744b87bd63c27f00c3f20e, 878595c0cf284b1e919266439baf64aa; no voters:
I20260810 18:09:26.914606 7711 raft_consensus.cc:2781] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:09:26.914867 7711 raft_consensus.cc:686] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [term 1 LEADER]: Becoming Leader. State: Replica: 3edfb4dd970b44ab919afc71b7cff75d, State: Running, Role: LEADER
I20260810 18:09:26.915225 7711 consensus_queue.cc:237] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } }
I20260810 18:09:26.918463 7166 catalog_manager.cc:5495] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66). New cstate: current_term: 1 leader_uuid: "3edfb4dd970b44ab919afc71b7cff75d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } health_report { overall_health: UNKNOWN } } }
I20260810 18:09:26.948493 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.944367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.21.182.66:41385)
0810 18:09:26.944540 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:26.944545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:26.944560 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:09:26.944744 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:26.944748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:26.944837 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:26.944970 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:09:26.945284 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.945834 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.946796 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:26.947041 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:26.947167 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:26.947554 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:09:26.948036 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:09:26.948203 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:26.948243 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 18:09:26.948307 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:09:26.954910 7628 raft_consensus.cc:1261] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa [term 1 FOLLOWER]: Refusing update from remote peer 3edfb4dd970b44ab919afc71b7cff75d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:09:26.955173 7328 raft_consensus.cc:1261] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Refusing update from remote peer 3edfb4dd970b44ab919afc71b7cff75d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:09:26.955830 7714 consensus_queue.cc:1035] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:09:26.956118 7711 consensus_queue.cc:1035] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:09:26.963183 7719 mvcc.cc:205] Tried to move back new op lower bound from 7317034463039188992 to 7317034462887677952. Current Snapshot: MvccSnapshot[applied={T|T < 7317034463039188992}]
I20260810 18:09:26.963395 7720 mvcc.cc:205] Tried to move back new op lower bound from 7317034463039188992 to 7317034462887677952. Current Snapshot: MvccSnapshot[applied={T|T < 7317034463039188992}]
I20260810 18:09:26.964555 7721 mvcc.cc:205] Tried to move back new op lower bound from 7317034463039188992 to 7317034462887677952. Current Snapshot: MvccSnapshot[applied={T|T < 7317034463039188992}]
I20260810 18:09:26.973222 7166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:09:26.973400 7166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:09:26.976107 7166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Sending DeleteTablet for 3 replicas of tablet 8d08370266b24a0e8691d56ff4b6ea5a
I20260810 18:09:26.977092 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 7236
I20260810 18:09:26.977077 7604 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d08370266b24a0e8691d56ff4b6ea5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:09:26 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:48760
I20260810 18:09:26.977289 7457 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d08370266b24a0e8691d56ff4b6ea5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:09:26 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:42874
I20260810 18:09:26.979652 7731 tablet_replica.cc:330] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d: stopping tablet replica
I20260810 18:09:26.980072 7731 raft_consensus.cc:2229] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:09:26.980547 7731 raft_consensus.cc:2258] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:09:26.980605 7730 tablet_replica.cc:330] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa: stopping tablet replica
I20260810 18:09:26.981025 7730 raft_consensus.cc:2229] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:09:26.981420 7730 raft_consensus.cc:2258] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:09:26.982311 7731 ts_tablet_manager.cc:1905] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:09:26.985042 7730 ts_tablet_manager.cc:1905] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:09:26.985412 7731 ts_tablet_manager.cc:1918] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:09:26.985515 7731 log.cc:1196] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/8d08370266b24a0e8691d56ff4b6ea5a
I20260810 18:09:26.985867 7731 ts_tablet_manager.cc:1939] T 8d08370266b24a0e8691d56ff4b6ea5a P 3edfb4dd970b44ab919afc71b7cff75d: Deleting consensus metadata
I20260810 18:09:26.987308 7151 catalog_manager.cc:4841] TS 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66:41385): tablet 8d08370266b24a0e8691d56ff4b6ea5a (table test-table [id=e68a1dd8ee4a4365961f1d13094a0c2b]) successfully deleted
I20260810 18:09:26.987846 7730 ts_tablet_manager.cc:1918] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:09:26.988001 7730 log.cc:1196] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/8d08370266b24a0e8691d56ff4b6ea5a
I20260810 18:09:26.988389 7730 ts_tablet_manager.cc:1939] T 8d08370266b24a0e8691d56ff4b6ea5a P 878595c0cf284b1e919266439baf64aa: Deleting consensus metadata
I20260810 18:09:26.989625 7149 catalog_manager.cc:4841] TS 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131): tablet 8d08370266b24a0e8691d56ff4b6ea5a (table test-table [id=e68a1dd8ee4a4365961f1d13094a0c2b]) successfully deleted
W20260810 18:09:26.991323 7151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:40959 (error 108)
I20260810 18:09:26.991655 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 7385
I20260810 18:09:26.997298 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:26.996875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:40959 (local address 127.0.0.1:47318)
0810 18:09:26.997066 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:26.997175 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 18:09:26.997539 7151 catalog_manager.cc:4584] TS 71db6faa5f744b87bd63c27f00c3f20e (127.21.182.65:40959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d08370266b24a0e8691d56ff4b6ea5a: Network error: Client connection negotiation failed: client connection to 127.21.182.65:40959: connect: Connection refused (error 111)
I20260810 18:09:27.003176 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 7533
I20260810 18:09:27.012979 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 7132
I20260810 18:09:47.023878 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:33733
--webserver_interface=127.21.182.126
--webserver_port=33425
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:09:47.150676 7733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:47.151080 7733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:47.151124 7733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:47.155577 7733 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 18:09:47.155663 7733 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 18:09:47.155680 7733 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 18:09:47.155694 7733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:09:47.155714 7733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:47.155738 7733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:47.155755 7733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:09:47.155771 7733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:09:47.161677 7733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:33733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=33425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:47.162969 7733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:47.164176 7733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:09:47.180294 7733 server_base.cc:1034] running on GCE node
W20260810 18:09:47.180313 7740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:47.180295 7741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:47.180605 7743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:47.181375 7733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:47.182876 7733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:47.184087 7733 hybrid_clock.cc:648] HybridClock initialized: now 1786385387184069 us; error 34 us; skew 500 ppm
Aug 10 18:09:47 dist-test-slave-27sr krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385387, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:47.194108 7733 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:09:47.195385 7733 webserver.cc:460] Webserver started at http://127.21.182.126:33425/ using document root <none> and password file <none>
I20260810 18:09:47.196223 7733 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:47.196379 7733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:47.201313 7733 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:09:47.204003 7750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:47.205495 7733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:47.205689 7733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "0967adc98d744d208e5b223ac1e5bfea"
format_stamp: "Formatted at 2026-08-10 18:09:25 on dist-test-slave-27sr"
server_key: "c6e7f64b79bf022671bc8ca1238d22ee"
server_key_iv: "cfa2a24bcab1c6a86dcfc654f9f4ab82"
server_key_version: "encryptionkey@0"
I20260810 18:09:47.206429 7733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:47.220875 7733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:47.224679 7733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:47.224951 7733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:47.233952 7733 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:33733
I20260810 18:09:47.234455 7809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:33733 every 8 connection(s)
I20260810 18:09:47.235338 7733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:09:47.240490 7810 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:09:47.242465 7810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Bootstrap starting.
I20260810 18:09:47.243420 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 7733
I20260810 18:09:47.244017 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:40959
--local_ip_for_outbound_sockets=127.21.182.65
--tserver_master_addrs=127.21.182.126:33733
--webserver_port=44319
--webserver_interface=127.21.182.65
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:09:47.266566 7810 log.cc:824] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:47.278352 7810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:09:47.278894 7810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Bootstrap complete.
I20260810 18:09:47.284873 7810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } }
I20260810 18:09:47.285440 7810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0967adc98d744d208e5b223ac1e5bfea, State: Initialized, Role: FOLLOWER
I20260810 18:09:47.286046 7810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } }
I20260810 18:09:47.286202 7810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:09:47.286289 7810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:09:47.286394 7810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:09:47.289238 7810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } }
I20260810 18:09:47.289673 7810 leader_election.cc:302] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0967adc98d744d208e5b223ac1e5bfea; no voters:
I20260810 18:09:47.290095 7810 leader_election.cc:288] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:09:47.290397 7814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:09:47.290834 7814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 2 LEADER]: Becoming Leader. State: Replica: 0967adc98d744d208e5b223ac1e5bfea, State: Running, Role: LEADER
I20260810 18:09:47.291317 7814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } }
I20260810 18:09:47.291656 7810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:09:47.293218 7815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0967adc98d744d208e5b223ac1e5bfea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } } }
I20260810 18:09:47.293218 7817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0967adc98d744d208e5b223ac1e5bfea. Latest consensus state: current_term: 2 leader_uuid: "0967adc98d744d208e5b223ac1e5bfea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } } }
I20260810 18:09:47.293397 7817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:47.293397 7815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: This master's current role is: LEADER
I20260810 18:09:47.297798 7830 catalog_manager.cc:1259] Loaded cluster ID: e2465b5083454415ad78cdacbb8fc767
I20260810 18:09:47.297930 7830 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: loading cluster ID for follower catalog manager: success
I20260810 18:09:47.300966 7830 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: acquiring CA information for follower catalog manager: success
I20260810 18:09:47.302884 7830 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:09:47.303100 7827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:09:47.305444 7827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:09:47.305959 7827 catalog_manager.cc:1259] Loaded cluster ID: e2465b5083454415ad78cdacbb8fc767
I20260810 18:09:47.306041 7827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:09:47.306658 7827 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:09:47.307152 7827 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Loaded TSK: 0
I20260810 18:09:47.328595 7827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Generated new TSK 1
I20260810 18:09:47.328933 7827 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:09:47.383065 7812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:47.383355 7812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:47.383431 7812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:47.387444 7812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:47.387534 7812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:47.387641 7812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:09:47.393352 7812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:40959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=44319
--tserver_master_addrs=127.21.182.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:47.394483 7812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:47.395565 7812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:47.406684 7839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:47.406720 7838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:47.406694 7841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:47.407171 7812 server_base.cc:1034] running on GCE node
I20260810 18:09:47.407608 7812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:47.408352 7812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:47.409544 7812 hybrid_clock.cc:648] HybridClock initialized: now 1786385387409517 us; error 45 us; skew 500 ppm
Aug 10 18:09:47 dist-test-slave-27sr krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385387, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:47.412581 7812 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:09:47.413748 7812 webserver.cc:460] Webserver started at http://127.21.182.65:44319/ using document root <none> and password file <none>
I20260810 18:09:47.414444 7812 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:47.414549 7812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:47.418849 7812 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:09:47.421433 7848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:47.422892 7812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:09:47.423091 7812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "71db6faa5f744b87bd63c27f00c3f20e"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "60aecb92391c73d1ffb7201d18281b6d"
server_key_iv: "bfde7873ad98854cc60ba5362f985ad9"
server_key_version: "encryptionkey@0"
I20260810 18:09:47.423823 7812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:47.461428 7812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:47.465332 7812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:47.465575 7812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:47.466275 7812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:47.467519 7856 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 18:09:47.472024 7812 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 18:09:47.472124 7812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260810 18:09:47.472221 7812 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 18:09:47.474197 7812 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 18:09:47.474273 7812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 18:09:47.474578 7856 tablet_bootstrap.cc:492] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: Bootstrap starting.
I20260810 18:09:47.485795 7812 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:40959
I20260810 18:09:47.485889 7977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:40959 every 8 connection(s)
I20260810 18:09:47.486927 7812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:09:47.494552 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 7812
I20260810 18:09:47.495078 7856 log.cc:824] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:47.495137 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:41385
--local_ip_for_outbound_sockets=127.21.182.66
--tserver_master_addrs=127.21.182.126:33733
--webserver_port=39183
--webserver_interface=127.21.182.66
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:09:47 dist-test-slave-27sr krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385387, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:47.504369 7856 tablet_bootstrap.cc:492] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:09:47.504933 7856 tablet_bootstrap.cc:492] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: Bootstrap complete.
I20260810 18:09:47.506439 7856 ts_tablet_manager.cc:1397] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.013s
I20260810 18:09:47.508960 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.489431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:56121 (local address 127.21.182.126:33733)
0810 18:09:47.489848 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:47.489870 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:47.490460 (+ 590us) server_negotiation.cc:407] Connection header received
0810 18:09:47.491738 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:47.491751 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:47.492212 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:47.492598 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:47.494319 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.496079 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.497780 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.498617 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.498854 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.501921 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:47.501929 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:47.501947 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:47.501986 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:47.504402 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:47.505155 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:47.505162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:47.505168 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:47.505357 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:47.505867 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:47.505873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:47.505878 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:47.506439 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:47.506582 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:47.507521 (+ 939us) server_negotiation.cc:299] Negotiation successful
0810 18:09:47.507868 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":151,"threads_started":1}
I20260810 18:09:47.509624 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.489506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:56121)
0810 18:09:47.490237 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:47.490288 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:47.491202 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:47.492777 (+ 1575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:47.492787 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:47.493369 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:47.494043 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:47.494067 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.496272 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.496275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:47.497619 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:47.497630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.498819 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.498823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:47.499019 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.499752 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:47.499772 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:47.501645 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:47.504569 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:47.504579 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:47.504597 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:47.504989 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:47.505495 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:47.505501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:47.505507 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:47.505735 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:47.506680 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:47.506690 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:47.507096 (+ 406us) client_negotiation.cc:769] Sending connection context
0810 18:09:47.507362 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 18:09:47.507835 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":203,"threads_started":1}
I20260810 18:09:47.509578 7856 raft_consensus.cc:348] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } }
I20260810 18:09:47.510200 7856 raft_consensus.cc:729] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71db6faa5f744b87bd63c27f00c3f20e, State: Initialized, Role: FOLLOWER
I20260810 18:09:47.510811 7856 consensus_queue.cc:260] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } }
I20260810 18:09:47.511859 7856 ts_tablet_manager.cc:1428] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260810 18:09:47.512581 7979 heartbeater.cc:347] Connected to a master server at 127.21.182.126:33733
I20260810 18:09:47.512897 7979 heartbeater.cc:464] Registering TS with master...
I20260810 18:09:47.513444 7979 heartbeater.cc:511] Master 127.21.182.126:33733 requested a full tablet report, sending...
I20260810 18:09:47.516455 7769 ts_manager.cc:194] Registered new tserver with Master: 71db6faa5f744b87bd63c27f00c3f20e (127.21.182.65:40959)
I20260810 18:09:47.520778 7769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:56121
I20260810 18:09:47.524132 7979 heartbeater.cc:503] Master 127.21.182.126:33733 was elected leader, sending a full tablet report...
Aug 10 18:09:47 dist-test-slave-27sr krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385387, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for kudu/127.21.182.65@KRBTEST.COM
I20260810 18:09:47.531396 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.520130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.21.182.65:40959)
0810 18:09:47.520580 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:47.520592 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:47.521048 (+ 456us) server_negotiation.cc:407] Connection header received
0810 18:09:47.521348 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:47.521353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:47.521526 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:47.521688 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:47.522351 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.523104 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.524053 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.524471 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.524601 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.527319 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:47.527329 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:47.527340 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:47.527365 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:47.529239 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:47.529819 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:47.529824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:47.529826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:47.529908 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:47.530231 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:47.530233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:47.530234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:47.530517 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:47.530623 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:47.531090 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 18:09:47.531240 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":351,"thread_start_us":122,"threads_started":1}
I20260810 18:09:47.531386 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.520082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:40959 (local address 127.0.0.1:34688)
0810 18:09:47.520858 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:47.520896 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:47.521148 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:47.521691 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:47.521697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:47.522104 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:47.522216 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:47.522223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.523226 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.523230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:47.523865 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:47.523874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.524615 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.524619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:47.524739 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.525415 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:47.525432 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:47.527148 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:47.529373 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:47.529383 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:47.529389 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:47.529696 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:47.529998 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:47.530002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:47.530004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:47.530106 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:47.530632 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:47.530638 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:47.530786 (+ 148us) client_negotiation.cc:769] Sending connection context
0810 18:09:47.530962 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 18:09:47.531187 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":124,"threads_started":1}
I20260810 18:09:47.532698 7901 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d08370266b24a0e8691d56ff4b6ea5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:09:26 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:34688
I20260810 18:09:47.533643 7988 tablet_replica.cc:330] stopping tablet replica
I20260810 18:09:47.533962 7988 raft_consensus.cc:2229] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:09:47.534236 7988 raft_consensus.cc:2258] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:09:47.535281 7988 ts_tablet_manager.cc:1905] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:09:47.539057 7988 ts_tablet_manager.cc:1918] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:09:47.539148 7988 log.cc:1196] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal/wals/8d08370266b24a0e8691d56ff4b6ea5a
I20260810 18:09:47.539440 7988 ts_tablet_manager.cc:1939] T 8d08370266b24a0e8691d56ff4b6ea5a P 71db6faa5f744b87bd63c27f00c3f20e: Deleting consensus metadata
I20260810 18:09:47.541618 7754 catalog_manager.cc:4841] TS 71db6faa5f744b87bd63c27f00c3f20e (127.21.182.65:40959): tablet 8d08370266b24a0e8691d56ff4b6ea5a (table test-table [id=e68a1dd8ee4a4365961f1d13094a0c2b]) successfully deleted
W20260810 18:09:47.628191 7984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:47.628583 7984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:47.628680 7984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:47.633092 7984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:47.633172 7984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:47.633236 7984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:09:47.638778 7984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:41385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=39183
--tserver_master_addrs=127.21.182.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:47.640082 7984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:47.641139 7984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:47.651666 7997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:47.651816 7984 server_base.cc:1034] running on GCE node
W20260810 18:09:47.651666 7995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:47.652055 7999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:47.652493 7984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:47.653271 7984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:47.654496 7984 hybrid_clock.cc:648] HybridClock initialized: now 1786385387654453 us; error 56 us; skew 500 ppm
Aug 10 18:09:47 dist-test-slave-27sr krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385387, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:47.657788 7984 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:09:47.658901 7984 webserver.cc:460] Webserver started at http://127.21.182.66:39183/ using document root <none> and password file <none>
I20260810 18:09:47.659543 7984 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:47.659667 7984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:47.664175 7984 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:47.666548 8007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:47.668036 7984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:09:47.668192 7984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "3edfb4dd970b44ab919afc71b7cff75d"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "2ee9aaef8d0afa78ea8b917eca40d1bd"
server_key_iv: "b149a8508bf269dbb55a829653c401ef"
server_key_version: "encryptionkey@0"
I20260810 18:09:47.668889 7984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:47.683385 7984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:47.687284 7984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:47.687525 7984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:47.688262 7984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:47.689271 7984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:09:47.689347 7984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:47.689424 7984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:09:47.689476 7984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:47.699941 7984 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:41385
I20260810 18:09:47.700022 8133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:41385 every 8 connection(s)
I20260810 18:09:47.701102 7984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:09:47.704409 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 7984
I20260810 18:09:47.704941 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:34131
--local_ip_for_outbound_sockets=127.21.182.67
--tserver_master_addrs=127.21.182.126:33733
--webserver_port=43847
--webserver_interface=127.21.182.67
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:09:47 dist-test-slave-27sr krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385387, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:47.720665 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.703337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:36619 (local address 127.21.182.126:33733)
0810 18:09:47.703525 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:47.703530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:47.705027 (+ 1497us) server_negotiation.cc:407] Connection header received
0810 18:09:47.706630 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:47.706637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:47.706741 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:47.706874 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:47.709080 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.709680 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.712028 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.712500 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.712613 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.716277 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:47.716280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:47.716282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:47.716312 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:47.718039 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:47.718681 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:47.718685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:47.718686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:47.718779 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:47.719263 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:47.719267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:47.719268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:47.719476 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:47.719601 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:47.720289 (+ 688us) server_negotiation.cc:299] Negotiation successful
0810 18:09:47.720467 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 18:09:47.721720 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.703810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:36619)
0810 18:09:47.704846 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:47.704904 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:47.706357 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:47.707178 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:47.707190 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:47.708006 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:47.708851 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:47.708881 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.709880 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.709888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:47.711801 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:47.711821 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.712731 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.712739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:47.713031 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.713811 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:47.713841 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:47.716045 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:47.718189 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:47.718197 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:47.718218 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:47.718529 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:47.718926 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:47.718930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:47.718932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:47.719122 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:47.719600 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:47.719611 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:47.719995 (+ 384us) client_negotiation.cc:769] Sending connection context
0810 18:09:47.720218 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 18:09:47.720616 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":129,"threads_started":1}
I20260810 18:09:47.723215 8134 heartbeater.cc:347] Connected to a master server at 127.21.182.126:33733
I20260810 18:09:47.723526 8134 heartbeater.cc:464] Registering TS with master...
I20260810 18:09:47.724216 8134 heartbeater.cc:511] Master 127.21.182.126:33733 requested a full tablet report, sending...
I20260810 18:09:47.725569 7769 ts_manager.cc:194] Registered new tserver with Master: 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66:41385)
I20260810 18:09:47.726574 7769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:36619
W20260810 18:09:47.835479 8137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:09:47.835795 8137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:09:47.835846 8137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:09:47.840008 8137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:09:47.840085 8137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:09:47.840149 8137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:09:47.845775 8137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:34131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=43847
--tserver_master_addrs=127.21.182.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:09:47.847124 8137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:09:47.848340 8137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:09:47.859862 8143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:09:47.859841 8142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:47.860033 8137 server_base.cc:1034] running on GCE node
W20260810 18:09:47.860347 8145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:09:47.860848 8137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:09:47.861660 8137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:09:47.862880 8137 hybrid_clock.cc:648] HybridClock initialized: now 1786385387862859 us; error 37 us; skew 500 ppm
Aug 10 18:09:47 dist-test-slave-27sr krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385387, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:09:47.866544 8137 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:09:47.867699 8137 webserver.cc:460] Webserver started at http://127.21.182.67:43847/ using document root <none> and password file <none>
I20260810 18:09:47.868436 8137 fs_manager.cc:359] Metadata directory not provided
I20260810 18:09:47.868522 8137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:09:47.873065 8137 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:09:47.875445 8153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:47.877022 8137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:09:47.877177 8137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "878595c0cf284b1e919266439baf64aa"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "ecc9d69adddfe12974f6229c851bd424"
server_key_iv: "149061d30ae5ee5ff00740bcfcf3e901"
server_key_version: "encryptionkey@0"
I20260810 18:09:47.877816 8137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:09:47.913064 8137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:09:47.917291 8137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:09:47.917557 8137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:09:47.918283 8137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:09:47.919277 8137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:09:47.919333 8137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:47.919405 8137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:09:47.919445 8137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:09:47.930348 8137 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:34131
I20260810 18:09:47.930398 8277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:34131 every 8 connection(s)
I20260810 18:09:47.931725 8137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:09:47.935031 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 8137
Aug 10 18:09:47 dist-test-slave-27sr krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385387, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:09:47.945817 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.935977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.21.182.126:33733)
0810 18:09:47.936300 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:47.936307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:47.936650 (+ 343us) server_negotiation.cc:407] Connection header received
0810 18:09:47.936768 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:47.936771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:47.936830 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:47.936943 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:47.937813 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.938724 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.939658 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.940050 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.940137 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.941963 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:47.941966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:47.941968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:47.941994 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:47.943800 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:47.944291 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:47.944295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:47.944297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:47.944408 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:47.944733 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:47.944737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:47.944738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:47.944984 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:47.945235 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:47.945403 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 18:09:47.945553 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260810 18:09:47.949105 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.934038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:32785 (local address 127.21.182.126:33733)
0810 18:09:47.934197 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:47.934202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:47.935815 (+ 1613us) server_negotiation.cc:407] Connection header received
0810 18:09:47.937271 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:47.937274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:47.937319 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:47.937389 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:47.939039 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.939577 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.941223 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.941440 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.941534 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.944474 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:47.944480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:47.944482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:47.944509 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:47.946241 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:47.947024 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:47.947027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:47.947029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:47.947123 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:47.947669 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:47.947672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:47.947673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:47.947954 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:47.948080 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:47.948849 (+ 769us) server_negotiation.cc:299] Negotiation successful
0810 18:09:47.948978 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:09:47.950160 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.934480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:32785)
0810 18:09:47.935613 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:47.935663 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:47.937005 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:47.937588 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:47.937595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:47.938129 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:09:47.938881 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:47.938910 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.939736 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.939739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:47.941077 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:47.941093 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.941621 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.941625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:47.941835 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.942504 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:09:47.942520 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:09:47.944296 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:09:47.946399 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:47.946411 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:47.946436 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:47.946903 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:47.947291 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:09:47.947295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:09:47.947297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:09:47.947511 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:09:47.948264 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:09:47.948271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:09:47.948564 (+ 293us) client_negotiation.cc:769] Sending connection context
0810 18:09:47.948751 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 18:09:47.949161 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":715,"threads_started":1}
I20260810 18:09:47.951529 8278 heartbeater.cc:347] Connected to a master server at 127.21.182.126:33733
I20260810 18:09:47.951857 8278 heartbeater.cc:464] Registering TS with master...
I20260810 18:09:47.952535 8278 heartbeater.cc:511] Master 127.21.182.126:33733 requested a full tablet report, sending...
I20260810 18:09:47.953799 7769 ts_manager.cc:194] Registered new tserver with Master: 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131)
I20260810 18:09:47.954509 7769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:32785
I20260810 18:09:47.960616 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:09:47.965301 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.961431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.21.182.126:33733)
0810 18:09:47.961578 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:47.961582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:47.961784 (+ 202us) server_negotiation.cc:407] Connection header received
0810 18:09:47.961949 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:47.961951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:47.962079 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:47.962177 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:09:47.962450 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.963021 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.964077 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.964309 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.964413 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.964772 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:09:47.964944 (+ 172us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 18:09:47.965183 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60736: token expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 18:09:47.965418 7981 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60736: token expired
W20260810 18:09:47.965866 7687 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.182.126:33733)
I20260810 18:09:47.974377 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.966314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.21.182.126:33733)
0810 18:09:47.966458 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:47.966461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:47.966476 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:09:47.966623 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:47.966626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:47.966682 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:47.966764 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:09:47.967486 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.968085 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.968938 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.969171 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.969279 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.970666 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:09:47.970670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:09:47.970673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:09:47.970703 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:09:47.972167 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:47.972640 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:47.972644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:47.972646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:47.972771 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:09:47.973195 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:09:47.973198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:09:47.973200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:09:47.973518 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:09:47.973678 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:47.973986 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 18:09:47.974122 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:09:47.977377 7769 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:09:47.981026 7769 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:09:47.990319 7901 tablet_service.cc:1467] Processing CreateTablet for tablet 73d068b5b10f4d5782528bc058c14fea (DEFAULT_TABLE table=test-table [id=bebf69bb3f0e41448a835a2cb2432a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:09:47.991084 7901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d068b5b10f4d5782528bc058c14fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:47.994174 7856 tablet_bootstrap.cc:492] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e: Bootstrap starting.
I20260810 18:09:47.995155 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.988673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:41385 (local address 127.0.0.1:45460)
0810 18:09:47.989042 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:47.989059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:47.989152 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:47.990300 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:47.990304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:47.990325 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:47.990464 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:47.990472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.992002 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.992007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:47.993460 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:47.993473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.994664 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.994668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:47.994776 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.994837 (+ 61us) client_negotiation.cc:769] Sending connection context
0810 18:09:47.994895 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 18:09:47.994980 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286}
I20260810 18:09:47.995182 7856 tablet_bootstrap.cc:654] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:47.995965 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.988856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.21.182.66:41385)
0810 18:09:47.989561 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:47.989580 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:47.989608 (+ 28us) server_negotiation.cc:407] Connection header received
0810 18:09:47.989830 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:47.989836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:47.990133 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:47.990327 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:47.990643 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.991821 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.993642 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.994515 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.995379 (+ 864us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.995575 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:47.995647 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 18:09:47.995742 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":180,"threads_started":1}
I20260810 18:09:47.996374 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.989066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34131 (local address 127.0.0.1:44286)
0810 18:09:47.989440 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:47.989454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:47.989550 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:47.990558 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:47.990561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:47.990583 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:47.990686 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:47.990692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.992655 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.992660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:47.993847 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:47.993856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.995848 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.995851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:47.995998 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.996026 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:09:47.996117 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 18:09:47.996196 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":172,"threads_started":1}
I20260810 18:09:47.996793 7856 tablet_bootstrap.cc:492] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e: No bootstrap required, opened a new log
I20260810 18:09:47.996901 7856 ts_tablet_manager.cc:1397] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:09:47.996899 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:47.989357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.21.182.67:34131)
0810 18:09:47.989908 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:47.989920 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:47.989945 (+ 25us) server_negotiation.cc:407] Connection header received
0810 18:09:47.990023 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:47.990028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:47.990227 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:47.990401 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:47.990803 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.992469 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.994523 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:47.995533 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:47.995642 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:47.995785 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:47.996583 (+ 798us) server_negotiation.cc:299] Negotiation successful
0810 18:09:47.996728 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":155,"threads_started":1}
I20260810 18:09:47.997354 7856 raft_consensus.cc:348] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:09:47.997490 7856 raft_consensus.cc:374] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:47.997529 7856 raft_consensus.cc:729] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71db6faa5f744b87bd63c27f00c3f20e, State: Initialized, Role: FOLLOWER
I20260810 18:09:47.997661 7856 consensus_queue.cc:260] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:09:47.997931 7856 ts_tablet_manager.cc:1428] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:09:47.999359 8062 tablet_service.cc:1467] Processing CreateTablet for tablet 73d068b5b10f4d5782528bc058c14fea (DEFAULT_TABLE table=test-table [id=bebf69bb3f0e41448a835a2cb2432a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:09:48.000623 8062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d068b5b10f4d5782528bc058c14fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:48.005363 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 73d068b5b10f4d5782528bc058c14fea (DEFAULT_TABLE table=test-table [id=bebf69bb3f0e41448a835a2cb2432a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:09:48.006454 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d068b5b10f4d5782528bc058c14fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:09:48.008615 8293 tablet_bootstrap.cc:492] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d: Bootstrap starting.
I20260810 18:09:48.011386 8293 tablet_bootstrap.cc:654] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:48.012286 8293 log.cc:824] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:48.014472 8296 tablet_bootstrap.cc:492] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa: Bootstrap starting.
I20260810 18:09:48.014892 8293 tablet_bootstrap.cc:492] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d: No bootstrap required, opened a new log
I20260810 18:09:48.015122 8293 ts_tablet_manager.cc:1397] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 18:09:48.016932 8296 tablet_bootstrap.cc:654] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa: Neither blocks nor log segments found. Creating new log.
I20260810 18:09:48.018051 8296 log.cc:824] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa: Log is configured to *not* fsync() on all Append() calls
I20260810 18:09:48.018321 8293 raft_consensus.cc:348] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:09:48.018654 8293 raft_consensus.cc:374] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:48.018740 8293 raft_consensus.cc:729] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3edfb4dd970b44ab919afc71b7cff75d, State: Initialized, Role: FOLLOWER
I20260810 18:09:48.019219 8293 consensus_queue.cc:260] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:09:48.019979 8134 heartbeater.cc:503] Master 127.21.182.126:33733 was elected leader, sending a full tablet report...
I20260810 18:09:48.020349 8293 ts_tablet_manager.cc:1428] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:09:48.020745 8296 tablet_bootstrap.cc:492] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa: No bootstrap required, opened a new log
I20260810 18:09:48.020969 8296 ts_tablet_manager.cc:1397] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:09:48.024462 8296 raft_consensus.cc:348] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:09:48.024753 8296 raft_consensus.cc:374] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:09:48.024837 8296 raft_consensus.cc:729] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878595c0cf284b1e919266439baf64aa, State: Initialized, Role: FOLLOWER
I20260810 18:09:48.025311 8296 consensus_queue.cc:260] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:09:48.026050 8278 heartbeater.cc:503] Master 127.21.182.126:33733 was elected leader, sending a full tablet report...
I20260810 18:09:48.026403 8296 ts_tablet_manager.cc:1428] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260810 18:09:48.045694 8298 raft_consensus.cc:482] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:09:48.046042 8298 raft_consensus.cc:504] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:09:48.047506 8298 leader_election.cc:288] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131), 71db6faa5f744b87bd63c27f00c3f20e (127.21.182.65:40959)
I20260810 18:09:48.052008 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:48.048081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:57127 (local address 127.21.182.67:34131)
0810 18:09:48.048237 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:48.048242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:48.048259 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:09:48.048326 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:48.048328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:48.048369 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:48.048441 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:48.048831 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:48.049351 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:48.050681 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:48.051323 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:48.051485 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:48.051545 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:48.051762 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 18:09:48.051825 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:09:48.052011 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:48.047741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34131 (local address 127.21.182.66:57127)
0810 18:09:48.047883 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:48.047928 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:48.048037 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:48.048489 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:48.048492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:48.048507 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:48.048647 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:48.048654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:48.049458 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:48.049461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:48.050534 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:48.050544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:48.051492 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:48.051496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:48.051623 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:48.051652 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 18:09:48.051749 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 18:09:48.051837 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":24}
I20260810 18:09:48.052798 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d068b5b10f4d5782528bc058c14fea" candidate_uuid: "3edfb4dd970b44ab919afc71b7cff75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878595c0cf284b1e919266439baf64aa" is_pre_election: true
I20260810 18:09:48.052910 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:48.047971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:40959 (local address 127.21.182.66:44621)
0810 18:09:48.048359 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:48.048372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:09:48.048489 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:09:48.049026 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:09:48.049029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:09:48.049047 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:09:48.049169 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:48.049175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:48.050180 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:48.050183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:48.051372 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:09:48.051386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:09:48.052506 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:09:48.052509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:09:48.052618 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:48.052644 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:09:48.052694 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 18:09:48.052757 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":1,"thread_start_us":120,"threads_started":1}
I20260810 18:09:48.053203 8228 raft_consensus.cc:2454] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3edfb4dd970b44ab919afc71b7cff75d in term 0.
I20260810 18:09:48.053591 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:48.048103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:44621 (local address 127.21.182.65:40959)
0810 18:09:48.048715 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:48.048721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:48.048737 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:09:48.048803 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:48.048805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:48.048860 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:48.049017 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:09:48.049289 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:48.050040 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:48.051500 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:48.052350 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:48.053115 (+ 765us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:48.053302 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:48.053344 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 18:09:48.053425 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":108,"threads_started":1}
I20260810 18:09:48.053896 8008 leader_election.cc:302] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3edfb4dd970b44ab919afc71b7cff75d, 878595c0cf284b1e919266439baf64aa; no voters:
I20260810 18:09:48.054174 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d068b5b10f4d5782528bc058c14fea" candidate_uuid: "3edfb4dd970b44ab919afc71b7cff75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71db6faa5f744b87bd63c27f00c3f20e" is_pre_election: true
I20260810 18:09:48.054280 8298 raft_consensus.cc:2781] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:09:48.054375 8298 raft_consensus.cc:482] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:09:48.054423 8298 raft_consensus.cc:3037] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:48.054553 7926 raft_consensus.cc:2454] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3edfb4dd970b44ab919afc71b7cff75d in term 0.
I20260810 18:09:48.055958 8298 raft_consensus.cc:504] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:09:48.056347 8298 leader_election.cc:288] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [CANDIDATE]: Term 1 election: Requested vote from peers 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131), 71db6faa5f744b87bd63c27f00c3f20e (127.21.182.65:40959)
I20260810 18:09:48.056725 7926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d068b5b10f4d5782528bc058c14fea" candidate_uuid: "3edfb4dd970b44ab919afc71b7cff75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71db6faa5f744b87bd63c27f00c3f20e"
I20260810 18:09:48.056881 7926 raft_consensus.cc:3037] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:48.056885 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d068b5b10f4d5782528bc058c14fea" candidate_uuid: "3edfb4dd970b44ab919afc71b7cff75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878595c0cf284b1e919266439baf64aa"
I20260810 18:09:48.057055 8228 raft_consensus.cc:3037] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:09:48.058039 7926 raft_consensus.cc:2454] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3edfb4dd970b44ab919afc71b7cff75d in term 1.
I20260810 18:09:48.058384 8011 leader_election.cc:302] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3edfb4dd970b44ab919afc71b7cff75d, 71db6faa5f744b87bd63c27f00c3f20e; no voters:
I20260810 18:09:48.058418 8228 raft_consensus.cc:2454] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3edfb4dd970b44ab919afc71b7cff75d in term 1.
I20260810 18:09:48.058591 8298 raft_consensus.cc:2781] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:09:48.058948 8298 raft_consensus.cc:686] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [term 1 LEADER]: Becoming Leader. State: Replica: 3edfb4dd970b44ab919afc71b7cff75d, State: Running, Role: LEADER
I20260810 18:09:48.059312 8298 consensus_queue.cc:237] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:09:48.063351 7768 catalog_manager.cc:5495] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66). New cstate: current_term: 1 leader_uuid: "3edfb4dd970b44ab919afc71b7cff75d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } health_report { overall_health: UNKNOWN } } }
I20260810 18:09:48.078924 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:48.074890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.21.182.66:41385)
0810 18:09:48.075038 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:09:48.075043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:09:48.075060 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:09:48.075189 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:09:48.075192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:09:48.075256 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:09:48.075385 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:09:48.075648 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:48.076313 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:48.077229 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:09:48.077451 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:09:48.077567 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:09:48.077918 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:09:48.078386 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:09:48.078535 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 18:09:48.078577 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 18:09:48.078658 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:09:48.085716 7926 raft_consensus.cc:1261] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Refusing update from remote peer 3edfb4dd970b44ab919afc71b7cff75d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:09:48.085740 8228 raft_consensus.cc:1261] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa [term 1 FOLLOWER]: Refusing update from remote peer 3edfb4dd970b44ab919afc71b7cff75d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:09:48.086508 8298 consensus_queue.cc:1035] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:09:48.086761 8303 consensus_queue.cc:1035] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:09:48.092785 8306 mvcc.cc:205] Tried to move back new op lower bound from 7317034549590921216 to 7317034549494497280. Current Snapshot: MvccSnapshot[applied={T|T < 7317034549590921216}]
I20260810 18:09:48.095939 8305 mvcc.cc:205] Tried to move back new op lower bound from 7317034549590921216 to 7317034549494497280. Current Snapshot: MvccSnapshot[applied={T|T < 7317034549590921216}]
I20260810 18:09:48.096832 8307 mvcc.cc:205] Tried to move back new op lower bound from 7317034549590921216 to 7317034549494497280. Current Snapshot: MvccSnapshot[applied={T|T < 7317034549590921216}]
I20260810 18:09:48.104905 7769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:09:48.105110 7769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:09:48.107692 7769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Sending DeleteTablet for 3 replicas of tablet 73d068b5b10f4d5782528bc058c14fea
I20260810 18:09:48.108567 7901 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d068b5b10f4d5782528bc058c14fea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:09:48 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:34688
I20260810 18:09:48.108538 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d068b5b10f4d5782528bc058c14fea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:09:48 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:44286
I20260810 18:09:48.108537 8062 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d068b5b10f4d5782528bc058c14fea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:09:48 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:45460
I20260810 18:09:48.109064 8318 tablet_replica.cc:330] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d: stopping tablet replica
I20260810 18:09:48.109318 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 7812
I20260810 18:09:48.121054 8318 raft_consensus.cc:2229] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:09:48.121771 8318 raft_consensus.cc:2258] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:09:48.122337 8316 tablet_replica.cc:330] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa: stopping tablet replica
I20260810 18:09:48.122864 8316 raft_consensus.cc:2229] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:09:48.123322 8316 raft_consensus.cc:2258] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:09:48.123812 8318 ts_tablet_manager.cc:1905] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:09:48.125182 8316 ts_tablet_manager.cc:1905] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:09:48.126858 8318 ts_tablet_manager.cc:1918] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:09:48.127053 8318 log.cc:1196] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/73d068b5b10f4d5782528bc058c14fea
I20260810 18:09:48.127499 8318 ts_tablet_manager.cc:1939] T 73d068b5b10f4d5782528bc058c14fea P 3edfb4dd970b44ab919afc71b7cff75d: Deleting consensus metadata
I20260810 18:09:48.128872 8316 ts_tablet_manager.cc:1918] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:09:48.129063 8316 log.cc:1196] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/73d068b5b10f4d5782528bc058c14fea
W20260810 18:09:48.129132 7754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:40959 (error 108)
I20260810 18:09:48.129429 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 7984
I20260810 18:09:48.129670 7754 catalog_manager.cc:4841] TS 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66:41385): tablet 73d068b5b10f4d5782528bc058c14fea (table test-table [id=bebf69bb3f0e41448a835a2cb2432a66]) successfully deleted
I20260810 18:09:48.133821 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:09:48.133433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:40959 (local address 127.0.0.1:34702)
0810 18:09:48.133672 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 18:09:48.133739 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260810 18:09:48.134038 7754 catalog_manager.cc:4584] TS 71db6faa5f744b87bd63c27f00c3f20e (127.21.182.65:40959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73d068b5b10f4d5782528bc058c14fea: Network error: Client connection negotiation failed: client connection to 127.21.182.65:40959: connect: Connection refused (error 111)
I20260810 18:09:48.136458 8316 ts_tablet_manager.cc:1939] T 73d068b5b10f4d5782528bc058c14fea P 878595c0cf284b1e919266439baf64aa: Deleting consensus metadata
I20260810 18:09:48.137890 7751 catalog_manager.cc:4841] TS 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131): tablet 73d068b5b10f4d5782528bc058c14fea (table test-table [id=bebf69bb3f0e41448a835a2cb2432a66]) successfully deleted
I20260810 18:09:48.142796 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 8137
I20260810 18:09:48.154531 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 7733
I20260810 18:10:08.166186 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:33733
--webserver_interface=127.21.182.126
--webserver_port=33425
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:08.292204 8320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:08.292582 8320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:08.292627 8320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:08.296933 8320 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 18:10:08.297012 8320 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 18:10:08.297029 8320 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 18:10:08.297043 8320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:08.297063 8320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:08.297082 8320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:08.297098 8320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:08.297114 8320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:08.302872 8320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:33733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=33425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:08.304198 8320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:08.305335 8320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:08.315574 8328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:08.315618 8326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:08.315641 8320 server_base.cc:1034] running on GCE node
W20260810 18:10:08.315587 8325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:08.316612 8320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:08.317932 8320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:08.319144 8320 hybrid_clock.cc:648] HybridClock initialized: now 1786385408319108 us; error 53 us; skew 500 ppm
Aug 10 18:10:08 dist-test-slave-27sr krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385408, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:08.328655 8320 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:10:08.329861 8320 webserver.cc:460] Webserver started at http://127.21.182.126:33425/ using document root <none> and password file <none>
I20260810 18:10:08.330523 8320 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:08.330601 8320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:08.335131 8320 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:10:08.337875 8335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:08.339596 8320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:10:08.339738 8320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "0967adc98d744d208e5b223ac1e5bfea"
format_stamp: "Formatted at 2026-08-10 18:09:25 on dist-test-slave-27sr"
server_key: "c6e7f64b79bf022671bc8ca1238d22ee"
server_key_iv: "cfa2a24bcab1c6a86dcfc654f9f4ab82"
server_key_version: "encryptionkey@0"
I20260810 18:10:08.340540 8320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:08.352634 8320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:08.356339 8320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:08.356530 8320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:08.364809 8320 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:33733
I20260810 18:10:08.365341 8397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:33733 every 8 connection(s)
I20260810 18:10:08.366207 8320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:08.371436 8398 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:10:08.373595 8398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Bootstrap starting.
I20260810 18:10:08.375563 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 8320
I20260810 18:10:08.376166 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:40959
--local_ip_for_outbound_sockets=127.21.182.65
--tserver_master_addrs=127.21.182.126:33733
--webserver_port=44319
--webserver_interface=127.21.182.65
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:10:08.395217 8398 log.cc:824] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:08.410235 8398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:10:08.410761 8398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Bootstrap complete.
I20260810 18:10:08.415632 8398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } }
I20260810 18:10:08.416200 8398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0967adc98d744d208e5b223ac1e5bfea, State: Initialized, Role: FOLLOWER
I20260810 18:10:08.416806 8398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } }
I20260810 18:10:08.416947 8398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:08.417049 8398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:08.417196 8398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:10:08.419628 8398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } }
I20260810 18:10:08.420107 8398 leader_election.cc:302] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0967adc98d744d208e5b223ac1e5bfea; no voters:
I20260810 18:10:08.420574 8398 leader_election.cc:288] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 18:10:08.420682 8402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:10:08.420930 8402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [term 3 LEADER]: Becoming Leader. State: Replica: 0967adc98d744d208e5b223ac1e5bfea, State: Running, Role: LEADER
I20260810 18:10:08.421562 8402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } }
I20260810 18:10:08.422261 8398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:08.423454 8404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0967adc98d744d208e5b223ac1e5bfea. Latest consensus state: current_term: 3 leader_uuid: "0967adc98d744d208e5b223ac1e5bfea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } } }
I20260810 18:10:08.423600 8404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:08.423919 8403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0967adc98d744d208e5b223ac1e5bfea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0967adc98d744d208e5b223ac1e5bfea" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33733 } } }
I20260810 18:10:08.424009 8403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:08.424045 8413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:08.430809 8413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:08.431767 8413 catalog_manager.cc:1259] Loaded cluster ID: e2465b5083454415ad78cdacbb8fc767
I20260810 18:10:08.431881 8413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:08.434186 8413 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:08.435097 8413 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Loaded TSK: 0
I20260810 18:10:08.435233 8413 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0967adc98d744d208e5b223ac1e5bfea: Loaded TSK: 1
F20260810 18:10:08.436115 8413 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f12d40119ad google::LogMessage::Fail() at ??:0
@ 0x7f12d401488a google::LogMessage::SendToLog() at ??:0
@ 0x7f12d40114de google::LogMessage::Flush() at ??:0
@ 0x7f12d4012a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f12d6cd518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f12d6ccdf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f12de97a7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f12de95499c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f12de955178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f12de996df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f12de9b582b std::function<>::operator()() at ??:0
@ 0x7f12de954b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f12de955f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f12de952579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f12de996640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f12d62633d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f12d6263c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f12d62653f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f12d6254512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f12d78986db start_thread at ??:0
@ 0x7f12d2b0f61f clone at ??:0
W20260810 18:10:08.548432 8400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:08.548731 8400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:08.548777 8400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:08.553082 8400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:08.553164 8400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:08.553228 8400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:10:08.559085 8400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:40959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=44319
--tserver_master_addrs=127.21.182.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:08.560470 8400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:08.561623 8400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:08.574298 8450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:08.574353 8447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:08.574465 8448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:08.575313 8400 server_base.cc:1034] running on GCE node
I20260810 18:10:08.576013 8400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:08.577034 8400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:08.578270 8400 hybrid_clock.cc:648] HybridClock initialized: now 1786385408578246 us; error 33 us; skew 500 ppm
Aug 10 18:10:08 dist-test-slave-27sr krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385408, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:08.582413 8400 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:10:08.583588 8400 webserver.cc:460] Webserver started at http://127.21.182.65:44319/ using document root <none> and password file <none>
I20260810 18:10:08.584342 8400 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:08.584419 8400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:08.589310 8400 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:10:08.592411 8457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:08.594105 8400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:10:08.594282 8400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "71db6faa5f744b87bd63c27f00c3f20e"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "60aecb92391c73d1ffb7201d18281b6d"
server_key_iv: "bfde7873ad98854cc60ba5362f985ad9"
server_key_version: "encryptionkey@0"
I20260810 18:10:08.594977 8400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:08.613517 8400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:08.617390 8400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:08.617620 8400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:08.618302 8400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:08.619566 8465 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 18:10:08.626395 8400 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 18:10:08.626528 8400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260810 18:10:08.626629 8400 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 18:10:08.628705 8400 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 18:10:08.628794 8400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260810 18:10:08.629187 8465 tablet_bootstrap.cc:492] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e: Bootstrap starting.
I20260810 18:10:08.640372 8400 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:40959
I20260810 18:10:08.640633 8583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:40959 every 8 connection(s)
I20260810 18:10:08.641556 8400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:10:08.646265 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 8400
I20260810 18:10:08.646879 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:41385
--local_ip_for_outbound_sockets=127.21.182.66
--tserver_master_addrs=127.21.182.126:33733
--webserver_port=39183
--webserver_interface=127.21.182.66
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:10:08.653988 8465 log.cc:824] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:08.666378 8465 tablet_bootstrap.cc:492] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:10:08.666910 8465 tablet_bootstrap.cc:492] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e: Bootstrap complete.
I20260810 18:10:08.668694 8465 ts_tablet_manager.cc:1397] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e: Time spent bootstrapping tablet: real 0.040s user 0.031s sys 0.008s
I20260810 18:10:08.672823 8465 raft_consensus.cc:348] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:10:08.673520 8465 raft_consensus.cc:729] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71db6faa5f744b87bd63c27f00c3f20e, State: Initialized, Role: FOLLOWER
I20260810 18:10:08.674170 8465 consensus_queue.cc:260] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:10:08.675567 8465 ts_tablet_manager.cc:1428] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
W20260810 18:10:08.712338 8587 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:10:08.643828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:51707)
0810 18:10:08.644464 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:08.644528 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:08.645425 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:08.711321 (+ 65896us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":174,"threads_started":1}
W20260810 18:10:08.713083 8585 heartbeater.cc:644] Failed to heartbeat to 127.21.182.126:33733 (0 consecutive failures): Network error: Failed to ping master at 127.21.182.126:33733: Client connection negotiation failed: client connection to 127.21.182.126:33733: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 18:10:08.714040 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:08.713704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:57313)
0810 18:10:08.713880 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:08.713974 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 18:10:08.714875 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:08.714632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:53231)
0810 18:10:08.714751 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:08.714804 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 18:10:08.715150 8585 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:10:08.791667 8589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:08.792078 8589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:08.792189 8589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:08.796639 8589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:08.796766 8589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:08.796866 8589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:10:08.802979 8589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:41385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=39183
--tserver_master_addrs=127.21.182.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:08.804347 8589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:08.805540 8589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:08.816577 8601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:08.816581 8599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:08.816577 8598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:08.817001 8589 server_base.cc:1034] running on GCE node
I20260810 18:10:08.817492 8589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:08.818279 8589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:08.819535 8589 hybrid_clock.cc:648] HybridClock initialized: now 1786385408819507 us; error 42 us; skew 500 ppm
Aug 10 18:10:08 dist-test-slave-27sr krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385408, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:08.823055 8589 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:10:08.824245 8589 webserver.cc:460] Webserver started at http://127.21.182.66:39183/ using document root <none> and password file <none>
I20260810 18:10:08.824924 8589 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:08.825032 8589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:08.829586 8589 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:10:08.832325 8608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:08.833921 8589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:10:08.834093 8589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "3edfb4dd970b44ab919afc71b7cff75d"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "2ee9aaef8d0afa78ea8b917eca40d1bd"
server_key_iv: "b149a8508bf269dbb55a829653c401ef"
server_key_version: "encryptionkey@0"
I20260810 18:10:08.834762 8589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:08.849474 8589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:08.853197 8589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:08.853461 8589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:08.854156 8589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:08.855119 8589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:08.855206 8589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:08.855310 8589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:08.855363 8589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:08.866027 8589 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:41385
I20260810 18:10:08.866062 8738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:41385 every 8 connection(s)
I20260810 18:10:08.867206 8589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:10:08.869931 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 8589
I20260810 18:10:08.870491 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:34131
--local_ip_for_outbound_sockets=127.21.182.67
--tserver_master_addrs=127.21.182.126:33733
--webserver_port=43847
--webserver_interface=127.21.182.67
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:10:08.872480 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:08.869838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:54911)
0810 18:10:08.870473 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:08.870645 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":183,"threads_started":1}
W20260810 18:10:08.873061 8739 heartbeater.cc:644] Failed to heartbeat to 127.21.182.126:33733 (0 consecutive failures): Network error: Failed to ping master at 127.21.182.126:33733: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
I20260810 18:10:08.873800 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:08.873553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:54461)
0810 18:10:08.873683 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:08.873738 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 18:10:08.874571 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:08.874278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:37435)
0810 18:10:08.874467 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:08.874517 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260810 18:10:08.874758 8739 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:10:09.001649 8742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:09.002025 8742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:09.002115 8742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:09.006635 8742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:09.006769 8742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:09.006870 8742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:10:09.013229 8742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:34131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=43847
--tserver_master_addrs=127.21.182.126:33733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:09.014483 8742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:09.015642 8742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:09.026396 8750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:09.026381 8748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:09.026382 8747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:09.026836 8742 server_base.cc:1034] running on GCE node
I20260810 18:10:09.027302 8742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:09.028105 8742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:09.029326 8742 hybrid_clock.cc:648] HybridClock initialized: now 1786385409029314 us; error 44 us; skew 500 ppm
Aug 10 18:10:09 dist-test-slave-27sr krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385409, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:09.032608 8742 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:10:09.033674 8742 webserver.cc:460] Webserver started at http://127.21.182.67:43847/ using document root <none> and password file <none>
I20260810 18:10:09.034394 8742 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:09.034510 8742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:09.039242 8742 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:10:09.041796 8758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:09.043195 8742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:10:09.043363 8742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "878595c0cf284b1e919266439baf64aa"
format_stamp: "Formatted at 2026-08-10 18:09:26 on dist-test-slave-27sr"
server_key: "ecc9d69adddfe12974f6229c851bd424"
server_key_iv: "149061d30ae5ee5ff00740bcfcf3e901"
server_key_version: "encryptionkey@0"
I20260810 18:10:09.044143 8742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:09.062644 8742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:09.066025 8742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:09.066262 8742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:09.066951 8742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:09.067865 8742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:09.067991 8742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:09.068069 8742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:09.068120 8742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:09.077946 8742 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:34131
I20260810 18:10:09.077972 8895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:34131 every 8 connection(s)
I20260810 18:10:09.079077 8742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:10:09.080092 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 8742
I20260810 18:10:09.083438 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:09.081567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:54835)
0810 18:10:09.082083 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:09.082196 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":129,"threads_started":1}
W20260810 18:10:09.084033 8896 heartbeater.cc:644] Failed to heartbeat to 127.21.182.126:33733 (0 consecutive failures): Network error: Failed to ping master at 127.21.182.126:33733: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
I20260810 18:10:09.084755 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:09.084525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:55723)
0810 18:10:09.084645 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:09.084695 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 18:10:09.085485 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:09.085277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:47941)
0810 18:10:09.085391 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:09.085434 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 18:10:09.085758 8896 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 18:10:09.716470 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:09.715963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:35947)
0810 18:10:09.716293 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:09.716358 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":81,"threads_started":1}
I20260810 18:10:09.876065 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:09.875448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:51719)
0810 18:10:09.875776 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:09.875852 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":80,"threads_started":1}
I20260810 18:10:10.087134 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:10.086614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:56849)
0810 18:10:10.086915 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:10.086982 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
I20260810 18:10:10.147390 8904 raft_consensus.cc:482] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:10:10.147683 8904 raft_consensus.cc:504] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:10:10.149101 8904 leader_election.cc:288] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66:41385), 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131)
Aug 10 18:10:10 dist-test-slave-27sr krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385408, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.67@KRBTEST.COM
Aug 10 18:10:10 dist-test-slave-27sr krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385408, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.66@KRBTEST.COM
I20260810 18:10:10.165380 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:10.149307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:41385 (local address 127.21.182.65:43403)
0810 18:10:10.149576 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:10.149595 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:10.149710 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:10.152052 (+ 2342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:10.152060 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:10.152710 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:10.153273 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:10.153300 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:10.155775 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:10.155781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:10.156368 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:10.156378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:10.157235 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:10.157240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:10.157361 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:10.157944 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:10:10.157964 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:10.160083 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:10.162925 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:10.162930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:10.162932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:10.163179 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:10.163595 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:10.163600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:10.163602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:10.163714 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:10.164514 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:10.164517 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:10.164689 (+ 172us) client_negotiation.cc:769] Sending connection context
0810 18:10:10.164917 (+ 228us) client_negotiation.cc:240] Negotiation successful
0810 18:10:10.165231 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":45,"spinlock_wait_cycles":2816}
I20260810 18:10:10.166023 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:10.149397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34131 (local address 127.21.182.65:47225)
0810 18:10:10.149951 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:10.149965 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:10.150066 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:10.152435 (+ 2369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:10.152437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:10.152709 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:10.153272 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:10.153293 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:10.154831 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:10.154834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:10.155689 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:10.155730 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:10.156852 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:10.156856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:10.157041 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:10.157779 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:10:10.157806 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:10.159705 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:10.162409 (+ 2704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:10.162415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:10.162430 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:10.162704 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:10.163161 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:10.163167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:10.163169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:10.163422 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:10.164341 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:10.164349 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:10.165592 (+ 1243us) client_negotiation.cc:769] Sending connection context
0810 18:10:10.165705 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 18:10:10.165860 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":958,"thread_start_us":100,"threads_started":1}
I20260810 18:10:10.166438 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:10.149856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:47225 (local address 127.21.182.67:34131)
0810 18:10:10.150323 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:10.150348 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:10.150387 (+ 39us) server_negotiation.cc:407] Connection header received
0810 18:10:10.151196 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:10.151204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:10.151545 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:10.151923 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:10.153489 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:10.154672 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:10.155856 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:10.156709 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:10.156898 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:10.160022 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:10.160032 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:10.160051 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:10.160088 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:10.162262 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:10.162833 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:10.162840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:10.162846 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:10.163041 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:10.163545 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:10.163549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:10.163551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:10.164191 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:10.164661 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:10.165944 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0810 18:10:10.166264 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":145,"threads_started":1}
I20260810 18:10:10.166786 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:10.149532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:43403 (local address 127.21.182.66:41385)
0810 18:10:10.149979 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:10.150013 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:10.150063 (+ 50us) server_negotiation.cc:407] Connection header received
0810 18:10:10.151043 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:10.151053 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:10.151546 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:10.151923 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:10.153497 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:10.155613 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:10.156504 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:10.157076 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:10.157281 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:10.160351 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:10.160361 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:10.160380 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:10.160420 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:10.162746 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:10.163309 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:10.163315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:10.163321 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:10.163450 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:10.163831 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:10.163834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:10.163836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:10.164396 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:10.166204 (+ 1808us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:10.166368 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 18:10:10.166657 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":74,"threads_started":1}
W20260810 18:10:10.169184 8458 leader_election.cc:341] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131): Not found: Tablet not found: 73d068b5b10f4d5782528bc058c14fea
W20260810 18:10:10.169431 8460 leader_election.cc:341] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66:41385): Not found: Tablet not found: 73d068b5b10f4d5782528bc058c14fea
I20260810 18:10:10.169543 8460 leader_election.cc:302] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71db6faa5f744b87bd63c27f00c3f20e; no voters: 3edfb4dd970b44ab919afc71b7cff75d, 878595c0cf284b1e919266439baf64aa
I20260810 18:10:10.169868 8904 raft_consensus.cc:2726] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:10:10.717893 8908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:10.717376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:47211)
0810 18:10:10.717677 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:10.717744 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
I20260810 18:10:10.877504 8909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:10.877014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:52369)
0810 18:10:10.877328 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:10.877396 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":87,"threads_started":1}
I20260810 18:10:11.088606 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:11.088134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:39103)
0810 18:10:11.088437 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:11.088502 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
I20260810 18:10:11.719337 8911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:11.718870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:55135)
0810 18:10:11.719171 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:11.719227 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":111,"threads_started":1}
I20260810 18:10:11.841774 8913 raft_consensus.cc:482] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:10:11.841949 8913 raft_consensus.cc:504] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:10:11.842453 8913 leader_election.cc:288] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66:41385), 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131)
W20260810 18:10:11.843318 8458 leader_election.cc:341] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131): Not found: Tablet not found: 73d068b5b10f4d5782528bc058c14fea
W20260810 18:10:11.843449 8460 leader_election.cc:341] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66:41385): Not found: Tablet not found: 73d068b5b10f4d5782528bc058c14fea
I20260810 18:10:11.843570 8460 leader_election.cc:302] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71db6faa5f744b87bd63c27f00c3f20e; no voters: 3edfb4dd970b44ab919afc71b7cff75d, 878595c0cf284b1e919266439baf64aa
I20260810 18:10:11.843745 8913 raft_consensus.cc:2726] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:10:11.878873 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:11.878399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:33225)
0810 18:10:11.878681 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:11.878746 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260810 18:10:12.090149 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:12.089506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:46089)
0810 18:10:12.089941 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:12.090050 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":139,"threads_started":1}
I20260810 18:10:12.720680 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:12.720238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:45699)
0810 18:10:12.720513 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:12.720577 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20260810 18:10:12.880210 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:12.879675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:53559)
0810 18:10:12.880031 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:12.880098 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":88,"threads_started":1}
I20260810 18:10:13.091658 8920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:13.091166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:47347)
0810 18:10:13.091489 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:13.091555 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":112,"threads_started":1}
I20260810 18:10:13.722272 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:13.721754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:53007)
0810 18:10:13.722086 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:13.722151 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":99,"threads_started":1}
I20260810 18:10:13.881475 8922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:13.881061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:53555)
0810 18:10:13.881314 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:13.881376 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":83,"threads_started":1}
I20260810 18:10:14.072999 8924 raft_consensus.cc:482] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:10:14.073140 8924 raft_consensus.cc:504] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:10:14.073575 8924 leader_election.cc:288] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66:41385), 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131)
W20260810 18:10:14.074429 8460 leader_election.cc:341] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66:41385): Not found: Tablet not found: 73d068b5b10f4d5782528bc058c14fea
W20260810 18:10:14.074540 8458 leader_election.cc:341] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131): Not found: Tablet not found: 73d068b5b10f4d5782528bc058c14fea
I20260810 18:10:14.074612 8458 leader_election.cc:302] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71db6faa5f744b87bd63c27f00c3f20e; no voters: 3edfb4dd970b44ab919afc71b7cff75d, 878595c0cf284b1e919266439baf64aa
I20260810 18:10:14.074893 8924 raft_consensus.cc:2726] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:10:14.093355 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:14.092663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:40965)
0810 18:10:14.093154 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:14.093251 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":313,"threads_started":1}
I20260810 18:10:14.723580 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:14.723144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:36215)
0810 18:10:14.723413 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:14.723474 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260810 18:10:14.882877 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:14.882390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:58367)
0810 18:10:14.882696 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:14.882767 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
I20260810 18:10:15.094729 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:15.094152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:48109)
0810 18:10:15.094535 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:15.094620 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":145,"threads_started":1}
I20260810 18:10:15.725052 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:15.724541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:43051)
0810 18:10:15.724876 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:15.724946 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":81,"threads_started":1}
I20260810 18:10:15.884483 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:15.883797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:33749)
0810 18:10:15.884281 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:15.884369 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":84,"threads_started":1}
I20260810 18:10:16.096271 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:16.095601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:41583)
0810 18:10:16.095998 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:16.096071 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":74,"threads_started":1}
I20260810 18:10:16.726343 8934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:16.725911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:59013)
0810 18:10:16.726177 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:16.726236 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260810 18:10:16.886135 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:16.885633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:47669)
0810 18:10:16.885928 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:16.885994 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":88,"threads_started":1}
I20260810 18:10:17.097890 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:17.097137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:53057)
0810 18:10:17.097621 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:17.097777 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":162,"threads_started":1}
I20260810 18:10:17.537210 8937 raft_consensus.cc:482] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:10:17.537389 8937 raft_consensus.cc:504] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:10:17.537967 8937 leader_election.cc:288] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66:41385), 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131)
W20260810 18:10:17.538897 8460 leader_election.cc:341] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66:41385): Not found: Tablet not found: 73d068b5b10f4d5782528bc058c14fea
W20260810 18:10:17.539108 8458 leader_election.cc:341] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131): Not found: Tablet not found: 73d068b5b10f4d5782528bc058c14fea
I20260810 18:10:17.539175 8458 leader_election.cc:302] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71db6faa5f744b87bd63c27f00c3f20e; no voters: 3edfb4dd970b44ab919afc71b7cff75d, 878595c0cf284b1e919266439baf64aa
I20260810 18:10:17.539371 8937 raft_consensus.cc:2726] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:10:17.727653 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:17.727202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:52307)
0810 18:10:17.727482 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:17.727546 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260810 18:10:17.887769 8939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:17.887198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:47877)
0810 18:10:17.887524 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:17.887599 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":92,"threads_started":1}
I20260810 18:10:18.099449 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:18.098849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:33889)
0810 18:10:18.099183 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:18.099257 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":85,"threads_started":1}
I20260810 18:10:18.729192 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:18.728697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:34667)
0810 18:10:18.729018 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:18.729085 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":108,"threads_started":1}
I20260810 18:10:18.889451 8942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:18.888859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:60891)
0810 18:10:18.889222 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:18.889293 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":89,"threads_started":1}
I20260810 18:10:19.101001 8943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:19.100522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:60075)
0810 18:10:19.100817 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:19.100889 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260810 18:10:19.730583 8944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:19.730087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:34797)
0810 18:10:19.730398 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:19.730475 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
I20260810 18:10:19.890985 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:19.890379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:39241)
0810 18:10:19.890759 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:19.890873 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":86,"threads_started":1}
I20260810 18:10:20.102437 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:20.101888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:47971)
0810 18:10:20.102255 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:20.102327 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":121,"threads_started":1}
I20260810 18:10:20.732065 8948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:20.731503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:54175)
0810 18:10:20.731802 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:20.731951 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":105,"threads_started":1}
I20260810 18:10:20.892508 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:20.891977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:49841)
0810 18:10:20.892321 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:20.892397 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":100,"threads_started":1}
I20260810 18:10:21.103860 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:21.103339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:59037)
0810 18:10:21.103646 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:21.103712 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":86,"threads_started":1}
I20260810 18:10:21.733590 8954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:21.733012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:57337)
0810 18:10:21.733366 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:21.733447 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":89,"threads_started":1}
I20260810 18:10:21.893864 8955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:21.893365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:33825)
0810 18:10:21.893664 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:21.893741 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":84,"threads_started":1}
I20260810 18:10:22.105335 8956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:22.104820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:60749)
0810 18:10:22.105137 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:22.105221 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":104,"threads_started":1}
I20260810 18:10:22.734968 8957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:22.734520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:54039)
0810 18:10:22.734806 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:22.734866 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260810 18:10:22.895360 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:22.894845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:43595)
0810 18:10:22.895182 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:22.895253 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260810 18:10:23.106763 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:23.106243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:47657)
0810 18:10:23.106534 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:23.106596 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
I20260810 18:10:23.312299 8960 raft_consensus.cc:482] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:10:23.312491 8960 raft_consensus.cc:504] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3edfb4dd970b44ab919afc71b7cff75d" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 41385 } } peers { permanent_uuid: "878595c0cf284b1e919266439baf64aa" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34131 } } peers { permanent_uuid: "71db6faa5f744b87bd63c27f00c3f20e" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 40959 } }
I20260810 18:10:23.313061 8960 leader_election.cc:288] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66:41385), 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131)
W20260810 18:10:23.313898 8460 leader_election.cc:341] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3edfb4dd970b44ab919afc71b7cff75d (127.21.182.66:41385): Not found: Tablet not found: 73d068b5b10f4d5782528bc058c14fea
W20260810 18:10:23.314009 8458 leader_election.cc:341] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 878595c0cf284b1e919266439baf64aa (127.21.182.67:34131): Not found: Tablet not found: 73d068b5b10f4d5782528bc058c14fea
I20260810 18:10:23.314136 8458 leader_election.cc:302] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71db6faa5f744b87bd63c27f00c3f20e; no voters: 3edfb4dd970b44ab919afc71b7cff75d, 878595c0cf284b1e919266439baf64aa
I20260810 18:10:23.314329 8960 raft_consensus.cc:2726] T 73d068b5b10f4d5782528bc058c14fea P 71db6faa5f744b87bd63c27f00c3f20e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:10:23.736689 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:23.736227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.65:56309)
0810 18:10:23.736524 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:23.736586 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":101,"threads_started":1}
I20260810 18:10:23.896879 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:23.896423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.66:32851)
0810 18:10:23.896711 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:23.896778 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":91,"threads_started":1}
I20260810 18:10:24.084864 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 8400
I20260810 18:10:24.100028 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 8589
I20260810 18:10:24.108594 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:24.108124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:33733 (local address 127.21.182.67:60013)
0810 18:10:24.108408 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:24.108484 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.126:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":94,"threads_started":1}
I20260810 18:10:24.111021 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 8742
2026-08-10T18:10:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59749 ms)
[----------] 30 tests from SecurityITest (259563 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:24 dist-test-slave-27sr krb5kdc[8968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:24 dist-test-slave-27sr krb5kdc[8968](info): set up 2 sockets
Aug 10 18:10:24 dist-test-slave-27sr krb5kdc[8968](info): commencing operation
krb5kdc: starting...
W20260810 18:10:25.418007 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.280s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:25 dist-test-slave-27sr krb5kdc[8968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:10:25.568948 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:42141
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:42141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:25.693357 8985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:25.693671 8985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:25.693715 8985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:25.698006 8985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:25.698074 8985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:25.698094 8985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:25.698110 8985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:25.698127 8985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:25.703827 8985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:42141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:42141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:25.705019 8985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:25.706125 8985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:10:25.716209 8985 server_base.cc:1034] running on GCE node
W20260810 18:10:25.716162 8991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:25.716192 8990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:25.716162 8993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:25.717128 8985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:25.718587 8985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:25.719821 8985 hybrid_clock.cc:648] HybridClock initialized: now 1786385425719802 us; error 32 us; skew 500 ppm
Aug 10 18:10:25 dist-test-slave-27sr krb5kdc[8968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385425, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:25.722945 8985 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:10:25.724041 8985 webserver.cc:460] Webserver started at http://127.21.182.126:39703/ using document root <none> and password file <none>
I20260810 18:10:25.724687 8985 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:25.724790 8985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:25.725111 8985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:25.727514 8985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "50e0d54e9732472a97c5ea3833364071"
format_stamp: "Formatted at 2026-08-10 18:10:25 on dist-test-slave-27sr"
server_key: "cd1f23d9421187e8c9ac98002ea5d4cd"
server_key_iv: "e1a660fa7f2404f253a14d339b4bfbdd"
server_key_version: "encryptionkey@0"
I20260810 18:10:25.728191 8985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "50e0d54e9732472a97c5ea3833364071"
format_stamp: "Formatted at 2026-08-10 18:10:25 on dist-test-slave-27sr"
server_key: "cd1f23d9421187e8c9ac98002ea5d4cd"
server_key_iv: "e1a660fa7f2404f253a14d339b4bfbdd"
server_key_version: "encryptionkey@0"
I20260810 18:10:25.733057 8985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260810 18:10:25.736105 9001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:25.737565 8985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:10:25.737723 8985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "50e0d54e9732472a97c5ea3833364071"
format_stamp: "Formatted at 2026-08-10 18:10:25 on dist-test-slave-27sr"
server_key: "cd1f23d9421187e8c9ac98002ea5d4cd"
server_key_iv: "e1a660fa7f2404f253a14d339b4bfbdd"
server_key_version: "encryptionkey@0"
I20260810 18:10:25.737851 8985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:25.748615 8985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:25.751716 8985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:25.752002 8985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:25.759967 8985 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:42141
I20260810 18:10:25.760025 9059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:42141 every 8 connection(s)
I20260810 18:10:25.760994 8985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:25.764055 9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:25.767705 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 8985
I20260810 18:10:25.767935 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:25.768227 22233 external_mini_cluster.cc:1428] Setting key e73509f3683badc2e386b22a048ffee7
I20260810 18:10:25.770532 9061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071: Bootstrap starting.
I20260810 18:10:25.773276 9061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:25.774243 9061 log.cc:824] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:25.776844 9061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071: No bootstrap required, opened a new log
Aug 10 18:10:25 dist-test-slave-27sr krb5kdc[8968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:10:25.781230 9061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e0d54e9732472a97c5ea3833364071" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42141 } }
I20260810 18:10:25.781553 9061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:25.781662 9061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e0d54e9732472a97c5ea3833364071, State: Initialized, Role: FOLLOWER
I20260810 18:10:25.782207 9061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e0d54e9732472a97c5ea3833364071" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42141 } }
I20260810 18:10:25.782403 9061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:25.782475 9061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:25.782578 9061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:25.783766 9061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e0d54e9732472a97c5ea3833364071" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42141 } }
I20260810 18:10:25.784210 9061 leader_election.cc:302] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50e0d54e9732472a97c5ea3833364071; no voters:
I20260810 18:10:25.784559 9061 leader_election.cc:288] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:25.784709 9066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:25.784777 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:25.769665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.21.182.126:42141)
0810 18:10:25.770096 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:25.770114 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:25.770151 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:10:25.770844 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:25.770864 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:25.771200 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:25.771490 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:25.772502 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:25.773550 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:25.774731 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:25.775404 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:25.775532 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:25.778541 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:25.778550 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:25.778570 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:25.778603 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:25.781307 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:25.781853 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:25.781859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:25.781864 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:25.782007 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:25.782291 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:25.782294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:25.782296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:25.782884 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:25.783080 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:25.783354 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 18:10:25.783684 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":138,"threads_started":1}
I20260810 18:10:25.784986 9066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [term 1 LEADER]: Becoming Leader. State: Replica: 50e0d54e9732472a97c5ea3833364071, State: Running, Role: LEADER
I20260810 18:10:25.785390 9066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e0d54e9732472a97c5ea3833364071" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42141 } }
I20260810 18:10:25.785899 9061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:25.786669 9067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50e0d54e9732472a97c5ea3833364071. Latest consensus state: current_term: 1 leader_uuid: "50e0d54e9732472a97c5ea3833364071" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e0d54e9732472a97c5ea3833364071" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42141 } } }
I20260810 18:10:25.786840 9067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:25.787189 9068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50e0d54e9732472a97c5ea3833364071" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e0d54e9732472a97c5ea3833364071" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42141 } } }
I20260810 18:10:25.787317 9068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:25.788277 9074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:25.792178 9074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:25.798105 9074 catalog_manager.cc:1347] Generated new cluster ID: 2ae4c13766584812a52745c8b499ebc5
I20260810 18:10:25.798199 9074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:25.809309 9074 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:25.810514 9074 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:25.818960 9074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071: Generated new TSK 0
I20260810 18:10:25.819763 9074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:10:25.863781 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:42141
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:25.988364 9088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:25.988616 9088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:25.988660 9088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:25.992699 9088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:25.992753 9088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:25.992810 9088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:10:25.997857 9088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:42141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:25.998826 9088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:25.999792 9088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:26.010402 9094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:26.010427 9088 server_base.cc:1034] running on GCE node
W20260810 18:10:26.010371 9098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:26.010371 9096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:26.011085 9088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:26.011838 9088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:26.013079 9088 hybrid_clock.cc:648] HybridClock initialized: now 1786385426013060 us; error 31 us; skew 500 ppm
Aug 10 18:10:26 dist-test-slave-27sr krb5kdc[8968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385426, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:26.016198 9088 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:10:26.017290 9088 webserver.cc:460] Webserver started at http://127.21.182.65:40325/ using document root <none> and password file <none>
I20260810 18:10:26.017966 9088 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:26.018067 9088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:26.018400 9088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:26.020654 9088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738"
format_stamp: "Formatted at 2026-08-10 18:10:26 on dist-test-slave-27sr"
server_key: "5be4fbacaf45bde1bec8e58d39096900"
server_key_iv: "46a466ab12fea3732f53e942345b0325"
server_key_version: "encryptionkey@0"
I20260810 18:10:26.021278 9088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738"
format_stamp: "Formatted at 2026-08-10 18:10:26 on dist-test-slave-27sr"
server_key: "5be4fbacaf45bde1bec8e58d39096900"
server_key_iv: "46a466ab12fea3732f53e942345b0325"
server_key_version: "encryptionkey@0"
I20260810 18:10:26.025400 9088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:10:26.028211 9106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:26.029644 9088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:26.029917 9088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738"
format_stamp: "Formatted at 2026-08-10 18:10:26 on dist-test-slave-27sr"
server_key: "5be4fbacaf45bde1bec8e58d39096900"
server_key_iv: "46a466ab12fea3732f53e942345b0325"
server_key_version: "encryptionkey@0"
I20260810 18:10:26.030054 9088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:26.046464 9088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:26.050388 9088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:26.050647 9088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:26.051386 9088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:26.052508 9088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:26.052591 9088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:26.052677 9088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:26.052726 9088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:26.063853 9088 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:39181
I20260810 18:10:26.063872 9235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:39181 every 8 connection(s)
I20260810 18:10:26.064942 9088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:10:26.072500 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 9088
I20260810 18:10:26.072626 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:10:26.072929 22233 external_mini_cluster.cc:1428] Setting key 71ced186856f97cb94e2cfa71323432a
Aug 10 18:10:26 dist-test-slave-27sr krb5kdc[8968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385426, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:10:26.081267 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.067050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:49659 (local address 127.21.182.126:42141)
0810 18:10:26.067202 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:26.067206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:26.068614 (+ 1408us) server_negotiation.cc:407] Connection header received
0810 18:10:26.069614 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:26.069618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:26.069684 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:26.069803 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:26.071147 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.071710 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.073083 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.073346 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.073483 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.076628 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:26.076635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:26.076641 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:26.076676 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:26.078329 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:26.079154 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:26.079158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:26.079159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:26.079263 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:26.079718 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:26.079721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:26.079722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:26.080084 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:26.080177 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:26.080985 (+ 808us) server_negotiation.cc:299] Negotiation successful
0810 18:10:26.081119 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:10:26.082742 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.067474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:42141 (local address 127.21.182.65:49659)
0810 18:10:26.068441 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:26.068487 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:26.069373 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:26.070042 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:26.070051 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:26.070451 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:26.070973 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.070987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.071851 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.071857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.072939 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.072951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.073559 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.073567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.073786 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.074531 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:10:26.074555 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:26.076434 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:26.078487 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:26.078496 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:26.078518 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:26.078989 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:26.079378 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:26.079382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:26.079384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:26.079601 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:26.080211 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:26.080221 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:26.080650 (+ 429us) client_negotiation.cc:769] Sending connection context
0810 18:10:26.080922 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 18:10:26.081365 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":593,"threads_started":1}
I20260810 18:10:26.084344 9236 heartbeater.cc:347] Connected to a master server at 127.21.182.126:42141
I20260810 18:10:26.084622 9236 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:26.085167 9236 heartbeater.cc:511] Master 127.21.182.126:42141 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
I20260810 18:10:26.087073 9018 ts_manager.cc:194] Registered new tserver with Master: 866c5de4b5dc4f9b9f5df36c2bf8d738 (127.21.182.65:39181)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
I20260810 18:10:26.088759 9018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:49659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:10:26.107149 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:42141
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:26.229249 9242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:26.229591 9242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:26.229638 9242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:26.234134 9242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:26.234236 9242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:26.234339 9242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:10:26.240564 9242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:42141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:26.241882 9242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:26.243059 9242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:26.254077 9249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:26.254065 9247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:26.254127 9242 server_base.cc:1034] running on GCE node
W20260810 18:10:26.254065 9251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:26.254868 9242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:26.255591 9242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:26.256760 9242 hybrid_clock.cc:648] HybridClock initialized: now 1786385426256740 us; error 34 us; skew 500 ppm
Aug 10 18:10:26 dist-test-slave-27sr krb5kdc[8968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385426, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:26.259830 9242 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:10:26.260869 9242 webserver.cc:460] Webserver started at http://127.21.182.66:35167/ using document root <none> and password file <none>
I20260810 18:10:26.261523 9242 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:26.261623 9242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:26.261893 9242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:26.264225 9242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "716a4f64cf084a59a922b269a020c6f4"
format_stamp: "Formatted at 2026-08-10 18:10:26 on dist-test-slave-27sr"
server_key: "c92ec81a9b745aa9c3d2a2a254f3b2a2"
server_key_iv: "cece229941fed5a76da284019ac72429"
server_key_version: "encryptionkey@0"
I20260810 18:10:26.264855 9242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "716a4f64cf084a59a922b269a020c6f4"
format_stamp: "Formatted at 2026-08-10 18:10:26 on dist-test-slave-27sr"
server_key: "c92ec81a9b745aa9c3d2a2a254f3b2a2"
server_key_iv: "cece229941fed5a76da284019ac72429"
server_key_version: "encryptionkey@0"
I20260810 18:10:26.269086 9242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:10:26.271997 9258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:26.273330 9242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:10:26.273491 9242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "716a4f64cf084a59a922b269a020c6f4"
format_stamp: "Formatted at 2026-08-10 18:10:26 on dist-test-slave-27sr"
server_key: "c92ec81a9b745aa9c3d2a2a254f3b2a2"
server_key_iv: "cece229941fed5a76da284019ac72429"
server_key_version: "encryptionkey@0"
I20260810 18:10:26.273623 9242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:26.299439 9242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:26.303087 9242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:26.303318 9242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:26.304033 9242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:26.305054 9242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:26.305131 9242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:26.305213 9242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:26.305267 9242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:26.314640 9242 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:37767
I20260810 18:10:26.314670 9391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:37767 every 8 connection(s)
I20260810 18:10:26.315588 9242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:10:26.316179 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 9242
I20260810 18:10:26.316314 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:10:26.316874 22233 external_mini_cluster.cc:1428] Setting key e304e230b15e7083e9f888887ed99888
Aug 10 18:10:26 dist-test-slave-27sr krb5kdc[8968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385426, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
I20260810 18:10:26.331138 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.317588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:44089 (local address 127.21.182.126:42141)
0810 18:10:26.317763 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:26.317769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:26.318750 (+ 981us) server_negotiation.cc:407] Connection header received
0810 18:10:26.320368 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:26.320374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:26.320440 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:26.320538 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:26.321871 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.322566 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.323720 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.324017 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.324134 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.327158 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:26.327161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:26.327163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:26.327190 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:26.328821 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:26.329406 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:26.329409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:26.329410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:26.329517 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:26.329888 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:26.329894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:26.329898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:26.330100 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:26.330211 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:26.330780 (+ 569us) server_negotiation.cc:299] Negotiation successful
0810 18:10:26.330956 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
I20260810 18:10:26.332121 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.318089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:42141 (local address 127.21.182.66:44089)
0810 18:10:26.318584 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:26.318654 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:26.320152 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:26.320740 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:26.320749 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:26.321199 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:26.321699 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.321713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.322689 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.322691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.323578 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.323587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.324183 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.324188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.324364 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.325043 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:10:26.325060 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:26.326968 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:26.328949 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:26.328954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:26.328967 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:26.329296 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:26.329621 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:26.329624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:26.329626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:26.329781 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:26.330211 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:26.330219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:26.330495 (+ 276us) client_negotiation.cc:769] Sending connection context
0810 18:10:26.330701 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 18:10:26.331074 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":144,"threads_started":1}
I20260810 18:10:26.333349 9392 heartbeater.cc:347] Connected to a master server at 127.21.182.126:42141
I20260810 18:10:26.333630 9392 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:26.334220 9392 heartbeater.cc:511] Master 127.21.182.126:42141 requested a full tablet report, sending...
I20260810 18:10:26.335575 9018 ts_manager.cc:194] Registered new tserver with Master: 716a4f64cf084a59a922b269a020c6f4 (127.21.182.66:37767)
I20260810 18:10:26.336254 9018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:44089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:10:26.352020 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:42141
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:26.475934 9397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:26.476303 9397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:26.476415 9397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:26.481244 9397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:26.481350 9397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:26.481456 9397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:10:26.487874 9397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:42141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:26.489182 9397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:26.490377 9397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:26.500958 9406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:26.500972 9404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:26.500958 9402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:26.501226 9397 server_base.cc:1034] running on GCE node
I20260810 18:10:26.501770 9397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:26.502404 9397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:26.503569 9397 hybrid_clock.cc:648] HybridClock initialized: now 1786385426503554 us; error 29 us; skew 500 ppm
Aug 10 18:10:26 dist-test-slave-27sr krb5kdc[8968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385426, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:26.514339 9397 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:10:26.515442 9397 webserver.cc:460] Webserver started at http://127.21.182.67:40193/ using document root <none> and password file <none>
I20260810 18:10:26.516168 9397 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:26.516244 9397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:26.516531 9397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:26.519091 9397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "1b18be82d9554dddacdb24169672c348"
format_stamp: "Formatted at 2026-08-10 18:10:26 on dist-test-slave-27sr"
server_key: "ed33665ac2e06a0dd9447089fdd51812"
server_key_iv: "228fd71c9ab9570db2b20dcdfdbcfdbc"
server_key_version: "encryptionkey@0"
I20260810 18:10:26.519659 9397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "1b18be82d9554dddacdb24169672c348"
format_stamp: "Formatted at 2026-08-10 18:10:26 on dist-test-slave-27sr"
server_key: "ed33665ac2e06a0dd9447089fdd51812"
server_key_iv: "228fd71c9ab9570db2b20dcdfdbcfdbc"
server_key_version: "encryptionkey@0"
I20260810 18:10:26.523695 9397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 18:10:26.526587 9415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:26.528056 9397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:26.528195 9397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "1b18be82d9554dddacdb24169672c348"
format_stamp: "Formatted at 2026-08-10 18:10:26 on dist-test-slave-27sr"
server_key: "ed33665ac2e06a0dd9447089fdd51812"
server_key_iv: "228fd71c9ab9570db2b20dcdfdbcfdbc"
server_key_version: "encryptionkey@0"
I20260810 18:10:26.528314 9397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:26.544718 9397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:26.548213 9397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:26.548437 9397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:26.549093 9397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:26.550086 9397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:26.550173 9397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:26.550275 9397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:26.550328 9397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:26.560756 9397 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:45521
I20260810 18:10:26.560779 9544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:45521 every 8 connection(s)
I20260810 18:10:26.561787 9397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:10:26.570086 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 9397
I20260810 18:10:26.570291 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:10:26.570631 22233 external_mini_cluster.cc:1428] Setting key c7194c70e8ca4027f36e5aa3d7ff3238
Aug 10 18:10:26 dist-test-slave-27sr krb5kdc[8968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385426, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:10:26.576858 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.563943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:52179 (local address 127.21.182.126:42141)
0810 18:10:26.564128 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:26.564133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:26.565075 (+ 942us) server_negotiation.cc:407] Connection header received
0810 18:10:26.566527 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:26.566531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:26.566591 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:26.566696 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:26.567974 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.568518 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.569866 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.570210 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.570342 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.573479 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:26.573482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:26.573484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:26.573511 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:26.574818 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:26.575299 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:26.575301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:26.575303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:26.575388 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:26.575708 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:26.575710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:26.575711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:26.575994 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:26.576088 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:26.576627 (+ 539us) server_negotiation.cc:299] Negotiation successful
0810 18:10:26.576732 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:10:26.577888 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.564347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:42141 (local address 127.21.182.67:52179)
0810 18:10:26.564898 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:26.564945 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:26.566289 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:26.566909 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:26.566916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:26.567305 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:26.567802 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.567817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.568632 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.568635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.569601 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.569612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.570421 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.570430 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.570576 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.571290 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:10:26.571312 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:26.573326 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:26.574925 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:26.574931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:26.574943 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:26.575206 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:26.575476 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:26.575478 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:26.575480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:26.575619 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:26.576126 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:26.576131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:26.576394 (+ 263us) client_negotiation.cc:769] Sending connection context
0810 18:10:26.576579 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 18:10:26.576862 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":123,"threads_started":1}
I20260810 18:10:26.579190 9545 heartbeater.cc:347] Connected to a master server at 127.21.182.126:42141
I20260810 18:10:26.579473 9545 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:26.580091 9545 heartbeater.cc:511] Master 127.21.182.126:42141 requested a full tablet report, sending...
I20260810 18:10:26.581213 9018 ts_manager.cc:194] Registered new tserver with Master: 1b18be82d9554dddacdb24169672c348 (127.21.182.67:45521)
I20260810 18:10:26.581835 9018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:52179
I20260810 18:10:26.585181 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:10:26.596335 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.587637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44864 (local address 127.21.182.126:42141)
0810 18:10:26.587764 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:26.587768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:26.588218 (+ 450us) server_negotiation.cc:407] Connection header received
0810 18:10:26.588354 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:26.588357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:26.588403 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:26.588484 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:26.589297 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.589962 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.590987 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.591208 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.591304 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.592909 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:26.592913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:26.592915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:26.592943 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:26.594438 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:26.595003 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:26.595007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:26.595008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:26.595106 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:26.595436 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:26.595438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:26.595440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:26.595695 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:26.595803 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:26.596091 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 18:10:26.596208 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:10:26.599673 9018 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:10:26.602938 9018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:10:26.619508 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.613066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:39181 (local address 127.0.0.1:57178)
0810 18:10:26.613345 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:26.613362 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:26.613474 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:26.614656 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:26.614663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:26.614686 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:26.614933 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.614943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.616601 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.616604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.617751 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.617762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.618886 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.618889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.619017 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.619059 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 18:10:26.619206 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 18:10:26.619346 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":60,"thread_start_us":116,"threads_started":1}
I20260810 18:10:26.619521 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.613564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.21.182.65:39181)
0810 18:10:26.613910 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:26.613920 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:26.613947 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:10:26.614031 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:26.614039 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:26.614286 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:26.614468 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:26.615075 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.616399 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.617957 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.618734 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.618864 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.619050 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:26.619242 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 18:10:26.619345 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":143,"threads_started":1}
I20260810 18:10:26.619838 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.613398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:37767 (local address 127.0.0.1:40194)
0810 18:10:26.613839 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:26.613853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:26.613971 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:26.614970 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:26.614977 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:26.615004 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:26.615109 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.615115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.616531 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.616535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.617958 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.617970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.619285 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.619289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.619410 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.619435 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:10:26.619493 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 18:10:26.619606 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":107,"thread_start_us":98,"threads_started":1}
I20260810 18:10:26.620461 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.612618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:45521 (local address 127.0.0.1:33602)
0810 18:10:26.613201 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:26.613253 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:26.613421 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:26.614696 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:26.614699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:26.614721 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:26.614937 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.614945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.616210 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.616218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.617958 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.617970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.619285 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.619289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.619869 (+ 580us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.620172 (+ 303us) client_negotiation.cc:769] Sending connection context
0810 18:10:26.620250 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 18:10:26.620336 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":617,"thread_start_us":112,"threads_started":1}
I20260810 18:10:26.620679 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.612757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.21.182.67:45521)
0810 18:10:26.614152 (+ 1395us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:26.614164 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:26.614195 (+ 31us) server_negotiation.cc:407] Connection header received
0810 18:10:26.614291 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:26.614299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:26.614523 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:26.614712 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:26.615093 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.616065 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.618128 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.619116 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.619343 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.619504 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:26.620293 (+ 789us) server_negotiation.cc:299] Negotiation successful
0810 18:10:26.620404 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1277,"thread_start_us":141,"threads_started":1}
I20260810 18:10:26.621080 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.613824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.21.182.66:37767)
0810 18:10:26.614458 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:26.614471 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:26.614494 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:10:26.614571 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:26.614576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:26.614753 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:26.614966 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:26.615253 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.616399 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.618128 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.619115 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.620654 (+ 1539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.620805 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:26.620857 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 18:10:26.620946 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":158,"threads_started":1}
I20260810 18:10:26.622936 9164 tablet_service.cc:1467] Processing CreateTablet for tablet 415973d892164b9eb2654e925e6021db (DEFAULT_TABLE table=test-table [id=6635350c8ed64506a201d7edf4113335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:26.624073 9164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415973d892164b9eb2654e925e6021db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:26.623799 9316 tablet_service.cc:1467] Processing CreateTablet for tablet 415973d892164b9eb2654e925e6021db (DEFAULT_TABLE table=test-table [id=6635350c8ed64506a201d7edf4113335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:26.623795 9468 tablet_service.cc:1467] Processing CreateTablet for tablet 415973d892164b9eb2654e925e6021db (DEFAULT_TABLE table=test-table [id=6635350c8ed64506a201d7edf4113335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:26.624974 9468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415973d892164b9eb2654e925e6021db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:26.624974 9316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415973d892164b9eb2654e925e6021db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:26.630888 9567 tablet_bootstrap.cc:492] T 415973d892164b9eb2654e925e6021db P 866c5de4b5dc4f9b9f5df36c2bf8d738: Bootstrap starting.
I20260810 18:10:26.632788 9569 tablet_bootstrap.cc:492] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4: Bootstrap starting.
I20260810 18:10:26.633651 9567 tablet_bootstrap.cc:654] T 415973d892164b9eb2654e925e6021db P 866c5de4b5dc4f9b9f5df36c2bf8d738: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:26.633658 9568 tablet_bootstrap.cc:492] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348: Bootstrap starting.
I20260810 18:10:26.634608 9567 log.cc:824] T 415973d892164b9eb2654e925e6021db P 866c5de4b5dc4f9b9f5df36c2bf8d738: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:26.635969 9568 tablet_bootstrap.cc:654] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:26.636848 9569 tablet_bootstrap.cc:654] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:26.637315 9567 tablet_bootstrap.cc:492] T 415973d892164b9eb2654e925e6021db P 866c5de4b5dc4f9b9f5df36c2bf8d738: No bootstrap required, opened a new log
I20260810 18:10:26.637377 9568 log.cc:824] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:26.637547 9567 ts_tablet_manager.cc:1397] T 415973d892164b9eb2654e925e6021db P 866c5de4b5dc4f9b9f5df36c2bf8d738: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260810 18:10:26.638083 9569 log.cc:824] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:26.639492 9568 tablet_bootstrap.cc:492] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348: No bootstrap required, opened a new log
I20260810 18:10:26.639698 9568 ts_tablet_manager.cc:1397] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260810 18:10:26.640197 9569 tablet_bootstrap.cc:492] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4: No bootstrap required, opened a new log
I20260810 18:10:26.640399 9569 ts_tablet_manager.cc:1397] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 18:10:26.642307 9567 raft_consensus.cc:348] T 415973d892164b9eb2654e925e6021db P 866c5de4b5dc4f9b9f5df36c2bf8d738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b18be82d9554dddacdb24169672c348" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45521 } } peers { permanent_uuid: "716a4f64cf084a59a922b269a020c6f4" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37767 } } peers { permanent_uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39181 } }
I20260810 18:10:26.642623 9567 raft_consensus.cc:374] T 415973d892164b9eb2654e925e6021db P 866c5de4b5dc4f9b9f5df36c2bf8d738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:26.642705 9567 raft_consensus.cc:729] T 415973d892164b9eb2654e925e6021db P 866c5de4b5dc4f9b9f5df36c2bf8d738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 866c5de4b5dc4f9b9f5df36c2bf8d738, State: Initialized, Role: FOLLOWER
I20260810 18:10:26.643268 9567 consensus_queue.cc:260] T 415973d892164b9eb2654e925e6021db P 866c5de4b5dc4f9b9f5df36c2bf8d738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b18be82d9554dddacdb24169672c348" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45521 } } peers { permanent_uuid: "716a4f64cf084a59a922b269a020c6f4" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37767 } } peers { permanent_uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39181 } }
I20260810 18:10:26.644019 9236 heartbeater.cc:503] Master 127.21.182.126:42141 was elected leader, sending a full tablet report...
I20260810 18:10:26.643914 9568 raft_consensus.cc:348] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b18be82d9554dddacdb24169672c348" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45521 } } peers { permanent_uuid: "716a4f64cf084a59a922b269a020c6f4" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37767 } } peers { permanent_uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39181 } }
I20260810 18:10:26.644177 9568 raft_consensus.cc:374] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:26.644238 9568 raft_consensus.cc:729] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b18be82d9554dddacdb24169672c348, State: Initialized, Role: FOLLOWER
I20260810 18:10:26.644359 9569 raft_consensus.cc:348] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b18be82d9554dddacdb24169672c348" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45521 } } peers { permanent_uuid: "716a4f64cf084a59a922b269a020c6f4" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37767 } } peers { permanent_uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39181 } }
I20260810 18:10:26.644603 9569 raft_consensus.cc:374] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:26.644660 9569 raft_consensus.cc:729] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716a4f64cf084a59a922b269a020c6f4, State: Initialized, Role: FOLLOWER
I20260810 18:10:26.644665 9568 consensus_queue.cc:260] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b18be82d9554dddacdb24169672c348" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45521 } } peers { permanent_uuid: "716a4f64cf084a59a922b269a020c6f4" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37767 } } peers { permanent_uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39181 } }
I20260810 18:10:26.645076 9569 consensus_queue.cc:260] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b18be82d9554dddacdb24169672c348" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45521 } } peers { permanent_uuid: "716a4f64cf084a59a922b269a020c6f4" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37767 } } peers { permanent_uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39181 } }
I20260810 18:10:26.645452 9545 heartbeater.cc:503] Master 127.21.182.126:42141 was elected leader, sending a full tablet report...
I20260810 18:10:26.645931 9568 ts_tablet_manager.cc:1428] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260810 18:10:26.645936 9569 ts_tablet_manager.cc:1428] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:10:26.646414 9392 heartbeater.cc:503] Master 127.21.182.126:42141 was elected leader, sending a full tablet report...
I20260810 18:10:26.647768 9567 ts_tablet_manager.cc:1428] T 415973d892164b9eb2654e925e6021db P 866c5de4b5dc4f9b9f5df36c2bf8d738: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260810 18:10:26.701925 9574 raft_consensus.cc:482] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:10:26.702244 9574 raft_consensus.cc:504] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b18be82d9554dddacdb24169672c348" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45521 } } peers { permanent_uuid: "716a4f64cf084a59a922b269a020c6f4" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37767 } } peers { permanent_uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39181 } }
I20260810 18:10:26.703615 9574 leader_election.cc:288] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 716a4f64cf084a59a922b269a020c6f4 (127.21.182.66:37767), 866c5de4b5dc4f9b9f5df36c2bf8d738 (127.21.182.65:39181)
I20260810 18:10:26.707711 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.704323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:44839 (local address 127.21.182.66:37767)
0810 18:10:26.704478 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:26.704482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:26.704494 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:10:26.704547 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:26.704550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:26.704590 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:26.704654 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:26.705002 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.705534 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.706695 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.707101 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.707252 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.707312 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:26.707507 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 18:10:26.707581 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:10:26.707711 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.703985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:37767 (local address 127.21.182.67:44839)
0810 18:10:26.704102 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:26.704119 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:26.704230 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:26.704708 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:26.704712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:26.704740 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:26.704851 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.704857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.705664 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.705671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.706545 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.706555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.707253 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.707256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.707372 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.707391 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:10:26.707485 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 18:10:26.707561 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":57}
I20260810 18:10:26.708216 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.704246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:41163 (local address 127.21.182.65:39181)
0810 18:10:26.704395 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:26.704400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:26.704990 (+ 590us) server_negotiation.cc:407] Connection header received
0810 18:10:26.705133 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:26.705136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:26.705186 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:26.705269 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:26.705564 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.706098 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.707195 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.707616 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.707750 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.707800 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:26.708038 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 18:10:26.708120 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:10:26.708604 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.703985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:39181 (local address 127.21.182.67:41163)
0810 18:10:26.704877 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:26.704889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:26.705013 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:26.705292 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:26.705295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:26.705312 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:26.705448 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.705454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.706219 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.706225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.707059 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:26.707069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.707746 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.707750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:26.707847 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.707871 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:10:26.708436 (+ 565us) client_negotiation.cc:240] Negotiation successful
0810 18:10:26.708498 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":65,"thread_start_us":155,"threads_started":1}
I20260810 18:10:26.708559 9340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415973d892164b9eb2654e925e6021db" candidate_uuid: "1b18be82d9554dddacdb24169672c348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716a4f64cf084a59a922b269a020c6f4" is_pre_election: true
I20260810 18:10:26.708931 9340 raft_consensus.cc:2454] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b18be82d9554dddacdb24169672c348 in term 0.
I20260810 18:10:26.709262 9184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415973d892164b9eb2654e925e6021db" candidate_uuid: "1b18be82d9554dddacdb24169672c348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738" is_pre_election: true
I20260810 18:10:26.709514 9416 leader_election.cc:302] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b18be82d9554dddacdb24169672c348, 716a4f64cf084a59a922b269a020c6f4; no voters:
I20260810 18:10:26.709580 9184 raft_consensus.cc:2454] T 415973d892164b9eb2654e925e6021db P 866c5de4b5dc4f9b9f5df36c2bf8d738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b18be82d9554dddacdb24169672c348 in term 0.
I20260810 18:10:26.709848 9574 raft_consensus.cc:2781] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:10:26.709959 9574 raft_consensus.cc:482] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:10:26.710021 9574 raft_consensus.cc:3037] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:26.711347 9574 raft_consensus.cc:504] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b18be82d9554dddacdb24169672c348" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45521 } } peers { permanent_uuid: "716a4f64cf084a59a922b269a020c6f4" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37767 } } peers { permanent_uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39181 } }
I20260810 18:10:26.711717 9574 leader_election.cc:288] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [CANDIDATE]: Term 1 election: Requested vote from peers 716a4f64cf084a59a922b269a020c6f4 (127.21.182.66:37767), 866c5de4b5dc4f9b9f5df36c2bf8d738 (127.21.182.65:39181)
I20260810 18:10:26.712108 9184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415973d892164b9eb2654e925e6021db" candidate_uuid: "1b18be82d9554dddacdb24169672c348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738"
I20260810 18:10:26.712146 9340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415973d892164b9eb2654e925e6021db" candidate_uuid: "1b18be82d9554dddacdb24169672c348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "716a4f64cf084a59a922b269a020c6f4"
I20260810 18:10:26.712280 9184 raft_consensus.cc:3037] T 415973d892164b9eb2654e925e6021db P 866c5de4b5dc4f9b9f5df36c2bf8d738 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:26.712284 9340 raft_consensus.cc:3037] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:26.713424 9184 raft_consensus.cc:2454] T 415973d892164b9eb2654e925e6021db P 866c5de4b5dc4f9b9f5df36c2bf8d738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b18be82d9554dddacdb24169672c348 in term 1.
I20260810 18:10:26.713649 9340 raft_consensus.cc:2454] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b18be82d9554dddacdb24169672c348 in term 1.
I20260810 18:10:26.713837 9419 leader_election.cc:302] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b18be82d9554dddacdb24169672c348, 866c5de4b5dc4f9b9f5df36c2bf8d738; no voters:
I20260810 18:10:26.714067 9574 raft_consensus.cc:2781] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:26.714334 9574 raft_consensus.cc:686] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [term 1 LEADER]: Becoming Leader. State: Replica: 1b18be82d9554dddacdb24169672c348, State: Running, Role: LEADER
I20260810 18:10:26.714740 9574 consensus_queue.cc:237] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b18be82d9554dddacdb24169672c348" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45521 } } peers { permanent_uuid: "716a4f64cf084a59a922b269a020c6f4" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37767 } } peers { permanent_uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39181 } }
I20260810 18:10:26.718907 9018 catalog_manager.cc:5495] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b18be82d9554dddacdb24169672c348 (127.21.182.67). New cstate: current_term: 1 leader_uuid: "1b18be82d9554dddacdb24169672c348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b18be82d9554dddacdb24169672c348" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "716a4f64cf084a59a922b269a020c6f4" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39181 } health_report { overall_health: UNKNOWN } } }
I20260810 18:10:26.745203 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.741382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.21.182.67:45521)
0810 18:10:26.741541 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:26.741545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:26.741568 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:10:26.741729 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:26.741732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:26.741795 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:26.741917 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:10:26.742205 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.742719 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.743604 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:26.743827 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:26.743999 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:26.744336 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:10:26.744774 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:10:26.744926 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:26.744965 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 18:10:26.745026 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:10:26.751739 9184 raft_consensus.cc:1261] T 415973d892164b9eb2654e925e6021db P 866c5de4b5dc4f9b9f5df36c2bf8d738 [term 1 FOLLOWER]: Refusing update from remote peer 1b18be82d9554dddacdb24169672c348: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:10:26.751739 9340 raft_consensus.cc:1261] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4 [term 1 FOLLOWER]: Refusing update from remote peer 1b18be82d9554dddacdb24169672c348: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:10:26.752774 9580 consensus_queue.cc:1035] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "866c5de4b5dc4f9b9f5df36c2bf8d738" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:10:26.753008 9574 consensus_queue.cc:1035] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "716a4f64cf084a59a922b269a020c6f4" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:10:26.763417 9584 mvcc.cc:205] Tried to move back new op lower bound from 7317034707966296064 to 7317034707826434048. Current Snapshot: MvccSnapshot[applied={T|T < 7317034707966296064 or (T in {7317034707966296064})}]
I20260810 18:10:26.765925 9586 mvcc.cc:205] Tried to move back new op lower bound from 7317034707966296064 to 7317034707826434048. Current Snapshot: MvccSnapshot[applied={T|T < 7317034707966296064 or (T in {7317034707966296064})}]
I20260810 18:10:26.770872 9585 mvcc.cc:205] Tried to move back new op lower bound from 7317034707966296064 to 7317034707826434048. Current Snapshot: MvccSnapshot[applied={T|T < 7317034707966296064}]
I20260810 18:10:26.770901 9013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:10:26.771091 9013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:10:26.773990 9013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50e0d54e9732472a97c5ea3833364071: Sending DeleteTablet for 3 replicas of tablet 415973d892164b9eb2654e925e6021db
I20260810 18:10:26.774940 9468 tablet_service.cc:1514] Processing DeleteTablet for tablet 415973d892164b9eb2654e925e6021db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:26 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:33602
I20260810 18:10:26.774988 9164 tablet_service.cc:1514] Processing DeleteTablet for tablet 415973d892164b9eb2654e925e6021db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:26 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:57178
I20260810 18:10:26.775096 9316 tablet_service.cc:1514] Processing DeleteTablet for tablet 415973d892164b9eb2654e925e6021db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:26 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:40194
I20260810 18:10:26.775658 9595 tablet_replica.cc:330] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348: stopping tablet replica
I20260810 18:10:26.776160 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 9088
I20260810 18:10:26.785559 9595 raft_consensus.cc:2229] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:10:26.786166 9595 raft_consensus.cc:2258] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:26.785964 9596 tablet_replica.cc:330] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4: stopping tablet replica
I20260810 18:10:26.786840 9596 raft_consensus.cc:2229] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:10:26.787406 9596 raft_consensus.cc:2258] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:26.789466 9595 ts_tablet_manager.cc:1905] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:10:26.789498 9596 ts_tablet_manager.cc:1905] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:10:26.792546 9595 ts_tablet_manager.cc:1918] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:10:26.792657 9595 log.cc:1196] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/415973d892164b9eb2654e925e6021db
I20260810 18:10:26.793007 9595 ts_tablet_manager.cc:1939] T 415973d892164b9eb2654e925e6021db P 1b18be82d9554dddacdb24169672c348: Deleting consensus metadata
I20260810 18:10:26.793732 9596 ts_tablet_manager.cc:1918] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:10:26.794028 9596 log.cc:1196] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/415973d892164b9eb2654e925e6021db
I20260810 18:10:26.794450 9596 ts_tablet_manager.cc:1939] T 415973d892164b9eb2654e925e6021db P 716a4f64cf084a59a922b269a020c6f4: Deleting consensus metadata
I20260810 18:10:26.794407 9002 catalog_manager.cc:4841] TS 1b18be82d9554dddacdb24169672c348 (127.21.182.67:45521): tablet 415973d892164b9eb2654e925e6021db (table test-table [id=6635350c8ed64506a201d7edf4113335]) successfully deleted
I20260810 18:10:26.795864 9002 catalog_manager.cc:4841] TS 716a4f64cf084a59a922b269a020c6f4 (127.21.182.66:37767): tablet 415973d892164b9eb2654e925e6021db (table test-table [id=6635350c8ed64506a201d7edf4113335]) successfully deleted
W20260810 18:10:26.796447 9005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:39181 (error 108)
I20260810 18:10:26.796643 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 9242
I20260810 18:10:26.798780 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:26.798427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:39181 (local address 127.0.0.1:57186)
0810 18:10:26.798588 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:26.798698 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:39181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 18:10:26.799022 9005 catalog_manager.cc:4584] TS 866c5de4b5dc4f9b9f5df36c2bf8d738 (127.21.182.65:39181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 415973d892164b9eb2654e925e6021db: Network error: Client connection negotiation failed: client connection to 127.21.182.65:39181: connect: Connection refused (error 111)
I20260810 18:10:26.806887 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 9397
I20260810 18:10:26.816859 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 8985
2026-08-10T18:10:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2706 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T18:10:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:26Z Disabled control of system clock
I20260810 18:10:26.855720 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:36767
--webserver_interface=127.21.182.126
--webserver_port=0
--builtin_ntp_servers=127.21.182.84:41261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:36767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:10:26.985311 9602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:26.985628 9602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:26.985674 9602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:26.989929 9602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:26.990010 9602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:26.990025 9602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:26.990043 9602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:26.990108 9602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:26.995381 9602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:41261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:36767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:36767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:26.996708 9602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:26.998291 9602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:10:27.008134 9602 server_base.cc:1034] running on GCE node
W20260810 18:10:27.008068 9608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:27.008078 9610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:27.008069 9612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:27.009120 9602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:27.010430 9602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:27.011638 9602 hybrid_clock.cc:648] HybridClock initialized: now 1786385427011614 us; error 44 us; skew 500 ppm
I20260810 18:10:27.014009 9602 webserver.cc:460] Webserver started at http://127.21.182.126:35765/ using document root <none> and password file <none>
I20260810 18:10:27.014739 9602 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:27.014811 9602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:27.015069 9602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:27.017536 9602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "18b01d206d384180be33b610a68b9ada"
format_stamp: "Formatted at 2026-08-10 18:10:27 on dist-test-slave-27sr"
server_key: "a980cf99f75322ff12f6b868d4486173"
server_key_iv: "1e6bde154fde95bd0de3d2d504ce16b0"
server_key_version: "encryptionkey@0"
I20260810 18:10:27.018232 9602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "18b01d206d384180be33b610a68b9ada"
format_stamp: "Formatted at 2026-08-10 18:10:27 on dist-test-slave-27sr"
server_key: "a980cf99f75322ff12f6b868d4486173"
server_key_iv: "1e6bde154fde95bd0de3d2d504ce16b0"
server_key_version: "encryptionkey@0"
I20260810 18:10:27.023159 9602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 18:10:27.026216 9619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:27.027642 9602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:10:27.027777 9602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "18b01d206d384180be33b610a68b9ada"
format_stamp: "Formatted at 2026-08-10 18:10:27 on dist-test-slave-27sr"
server_key: "a980cf99f75322ff12f6b868d4486173"
server_key_iv: "1e6bde154fde95bd0de3d2d504ce16b0"
server_key_version: "encryptionkey@0"
I20260810 18:10:27.027944 9602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:27.046442 9602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:27.047331 9602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:27.047583 9602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:27.056103 9602 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:36767
I20260810 18:10:27.056136 9679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:36767 every 8 connection(s)
I20260810 18:10:27.057184 9602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:27.060906 9681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:27.063738 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 9602
I20260810 18:10:27.063941 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:27.064342 22233 external_mini_cluster.cc:1428] Setting key 83aae5b3dd7908d538dc9242fe624b59
I20260810 18:10:27.068614 9681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada: Bootstrap starting.
I20260810 18:10:27.071214 9681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:27.072184 9681 log.cc:824] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:27.074702 9681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada: No bootstrap required, opened a new log
I20260810 18:10:27.075248 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.066210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.21.182.126:36767)
0810 18:10:27.067016 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:27.067034 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:27.067076 (+ 42us) server_negotiation.cc:407] Connection header received
0810 18:10:27.067761 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:27.067780 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:27.068309 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:27.068687 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:27.069158 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.070154 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.071102 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.071668 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.071865 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.072961 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:27.072966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:27.072981 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:27.073435 (+ 454us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:27.073498 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:27.073520 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:27.073618 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:27.073884 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 18:10:27.074255 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":181,"threads_started":1}
I20260810 18:10:27.079083 9681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b01d206d384180be33b610a68b9ada" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 36767 } }
I20260810 18:10:27.079371 9681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:27.079434 9681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b01d206d384180be33b610a68b9ada, State: Initialized, Role: FOLLOWER
I20260810 18:10:27.080040 9681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b01d206d384180be33b610a68b9ada" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 36767 } }
I20260810 18:10:27.080219 9681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:27.080313 9681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:27.080430 9681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:27.081532 9681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b01d206d384180be33b610a68b9ada" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 36767 } }
I20260810 18:10:27.081938 9681 leader_election.cc:302] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18b01d206d384180be33b610a68b9ada; no voters:
I20260810 18:10:27.082376 9681 leader_election.cc:288] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:27.082492 9688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:27.082849 9688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [term 1 LEADER]: Becoming Leader. State: Replica: 18b01d206d384180be33b610a68b9ada, State: Running, Role: LEADER
I20260810 18:10:27.083210 9688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b01d206d384180be33b610a68b9ada" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 36767 } }
I20260810 18:10:27.083873 9681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:27.085394 9690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18b01d206d384180be33b610a68b9ada. Latest consensus state: current_term: 1 leader_uuid: "18b01d206d384180be33b610a68b9ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b01d206d384180be33b610a68b9ada" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 36767 } } }
I20260810 18:10:27.085536 9690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:27.085883 9689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18b01d206d384180be33b610a68b9ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b01d206d384180be33b610a68b9ada" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 36767 } } }
I20260810 18:10:27.086000 9689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:27.086139 9701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:27.089843 9701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:27.097110 9701 catalog_manager.cc:1347] Generated new cluster ID: 4591dd03ee1c44058064f8cb983546bb
I20260810 18:10:27.097224 9701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:27.112161 9701 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:27.113695 9701 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:27.122066 9701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada: Generated new TSK 0
I20260810 18:10:27.123093 9701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:27.133785 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:36767
--builtin_ntp_servers=127.21.182.84:41261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:10:27.265440 9711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:27.265720 9711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:27.265761 9711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:27.269886 9711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:27.269949 9711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:27.270005 9711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:10:27.274914 9711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:41261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:36767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:27.276191 9711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:27.277668 9711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:27.288550 9716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:27.288576 9717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:27.288691 9711 server_base.cc:1034] running on GCE node
W20260810 18:10:27.288892 9719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:27.289351 9711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:27.290091 9711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:27.291268 9711 hybrid_clock.cc:648] HybridClock initialized: now 1786385427291225 us; error 58 us; skew 500 ppm
I20260810 18:10:27.293469 9711 webserver.cc:460] Webserver started at http://127.21.182.65:46187/ using document root <none> and password file <none>
I20260810 18:10:27.294122 9711 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:27.294225 9711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:27.294497 9711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:27.296867 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "14d519be7b2d435f9489e538d7705a01"
format_stamp: "Formatted at 2026-08-10 18:10:27 on dist-test-slave-27sr"
server_key: "e77c61092651b57813c27763d6879cc6"
server_key_iv: "04732c771ad12da89692a9b6090a0057"
server_key_version: "encryptionkey@0"
I20260810 18:10:27.297483 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "14d519be7b2d435f9489e538d7705a01"
format_stamp: "Formatted at 2026-08-10 18:10:27 on dist-test-slave-27sr"
server_key: "e77c61092651b57813c27763d6879cc6"
server_key_iv: "04732c771ad12da89692a9b6090a0057"
server_key_version: "encryptionkey@0"
I20260810 18:10:27.301748 9711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:10:27.304541 9726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:27.305936 9711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:27.306105 9711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "14d519be7b2d435f9489e538d7705a01"
format_stamp: "Formatted at 2026-08-10 18:10:27 on dist-test-slave-27sr"
server_key: "e77c61092651b57813c27763d6879cc6"
server_key_iv: "04732c771ad12da89692a9b6090a0057"
server_key_version: "encryptionkey@0"
I20260810 18:10:27.306246 9711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:27.322618 9711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:27.323443 9711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:27.323662 9711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:27.324450 9711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:27.325519 9711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:27.325596 9711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:27.325676 9711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:27.325722 9711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:27.335170 9711 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:34185
I20260810 18:10:27.335199 9855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:34185 every 8 connection(s)
I20260810 18:10:27.336283 9711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:10:27.343225 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 9711
I20260810 18:10:27.343374 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:10:27.343711 22233 external_mini_cluster.cc:1428] Setting key cd564b230c7b9f5239e85d49fcadb6ec
I20260810 18:10:27.345854 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:36767
--builtin_ntp_servers=127.21.182.84:41261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:10:27.347071 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.338432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:53203 (local address 127.21.182.126:36767)
0810 18:10:27.338678 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:27.338748 (+ 70us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:27.339945 (+ 1197us) server_negotiation.cc:407] Connection header received
0810 18:10:27.341493 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:27.341496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:27.341552 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:27.341623 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:27.342644 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.343446 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.344801 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.345044 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.345177 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.346512 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:27.346516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:27.346519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:27.346540 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:27.346561 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:27.346570 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:27.346656 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:27.346838 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 18:10:27.346920 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 18:10:27.348435 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.338871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:36767 (local address 127.21.182.65:53203)
0810 18:10:27.339734 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:27.339794 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:27.341225 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:27.341832 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:27.341840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:27.341936 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:27.342489 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.342504 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.343596 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.343599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.344657 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.344668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.345228 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.345232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.345420 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.346187 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:10:27.346189 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:27.346236 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:10:27.346316 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:27.346658 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:27.346672 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:27.346681 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:10:27.346838 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 18:10:27.347205 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":176,"threads_started":1}
I20260810 18:10:27.350247 9856 heartbeater.cc:347] Connected to a master server at 127.21.182.126:36767
I20260810 18:10:27.350541 9856 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:27.351289 9856 heartbeater.cc:511] Master 127.21.182.126:36767 requested a full tablet report, sending...
I20260810 18:10:27.353518 9631 ts_manager.cc:194] Registered new tserver with Master: 14d519be7b2d435f9489e538d7705a01 (127.21.182.65:34185)
I20260810 18:10:27.355319 9631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.182.65:53203
W20260810 18:10:27.471092 9859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:27.471414 9859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:27.471457 9859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:27.475631 9859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:27.475704 9859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:27.475764 9859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:10:27.480692 9859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:41261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:36767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:27.481905 9859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:27.483415 9859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:27.494752 9864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:27.494755 9865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:27.494776 9867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:27.495000 9859 server_base.cc:1034] running on GCE node
I20260810 18:10:27.495612 9859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:27.496411 9859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:27.497607 9859 hybrid_clock.cc:648] HybridClock initialized: now 1786385427497591 us; error 30 us; skew 500 ppm
I20260810 18:10:27.499598 9859 webserver.cc:460] Webserver started at http://127.21.182.66:41827/ using document root <none> and password file <none>
I20260810 18:10:27.500304 9859 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:27.500404 9859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:27.500636 9859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:27.502909 9859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "c9b1baa75ded4b4c93e14bb1cb291d97"
format_stamp: "Formatted at 2026-08-10 18:10:27 on dist-test-slave-27sr"
server_key: "72a3b9d8a60ed2928fb5c83874fd3d98"
server_key_iv: "6ba35b7562f9c5ac68e3626e6aa0d7e1"
server_key_version: "encryptionkey@0"
I20260810 18:10:27.503542 9859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "c9b1baa75ded4b4c93e14bb1cb291d97"
format_stamp: "Formatted at 2026-08-10 18:10:27 on dist-test-slave-27sr"
server_key: "72a3b9d8a60ed2928fb5c83874fd3d98"
server_key_iv: "6ba35b7562f9c5ac68e3626e6aa0d7e1"
server_key_version: "encryptionkey@0"
I20260810 18:10:27.507635 9859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 18:10:27.510525 9874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:27.512485 9859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:27.512728 9859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "c9b1baa75ded4b4c93e14bb1cb291d97"
format_stamp: "Formatted at 2026-08-10 18:10:27 on dist-test-slave-27sr"
server_key: "72a3b9d8a60ed2928fb5c83874fd3d98"
server_key_iv: "6ba35b7562f9c5ac68e3626e6aa0d7e1"
server_key_version: "encryptionkey@0"
I20260810 18:10:27.512887 9859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:27.528890 9859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:27.529939 9859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:27.530202 9859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:27.530930 9859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:27.532245 9859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:27.532326 9859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:27.532397 9859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:27.532441 9859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:27.541975 9859 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:43461
I20260810 18:10:27.541998 10009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:43461 every 8 connection(s)
I20260810 18:10:27.543011 9859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:10:27.545471 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 9859
I20260810 18:10:27.545603 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:10:27.545922 22233 external_mini_cluster.cc:1428] Setting key 588993f28c24f8b8a59fe2125ed717b2
I20260810 18:10:27.548136 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:36767
--builtin_ntp_servers=127.21.182.84:41261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:10:27.554544 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.545245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:32791 (local address 127.21.182.126:36767)
0810 18:10:27.545443 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:27.545449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:27.546784 (+ 1335us) server_negotiation.cc:407] Connection header received
0810 18:10:27.548394 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:27.548397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:27.548448 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:27.548493 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:27.549628 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.550408 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.552078 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.552516 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.552655 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.553880 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:27.553885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:27.553890 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:27.553912 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:27.553941 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:27.553954 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:27.554049 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:27.554208 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 18:10:27.554303 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 18:10:27.555370 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.545779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:36767 (local address 127.21.182.66:32791)
0810 18:10:27.546622 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:27.546678 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:27.548117 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:27.548846 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:27.548854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:27.548944 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:27.549463 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.549478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.550572 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.550578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.551938 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.551954 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.552692 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.552695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.552878 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.553584 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:10:27.553586 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:27.553630 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:10:27.553703 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:27.554054 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:27.554062 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:27.554066 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:10:27.554170 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 18:10:27.554449 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":192,"threads_started":1}
I20260810 18:10:27.556725 10011 heartbeater.cc:347] Connected to a master server at 127.21.182.126:36767
I20260810 18:10:27.557032 10011 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:27.557799 10011 heartbeater.cc:511] Master 127.21.182.126:36767 requested a full tablet report, sending...
I20260810 18:10:27.559159 9631 ts_manager.cc:194] Registered new tserver with Master: c9b1baa75ded4b4c93e14bb1cb291d97 (127.21.182.66:43461)
I20260810 18:10:27.559826 9631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.182.66:32791
W20260810 18:10:27.670567 10015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:27.670862 10015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:27.670909 10015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:27.675145 10015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:27.675220 10015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:27.675275 10015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:10:27.680285 10015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:41261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:36767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:27.681510 10015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:27.683018 10015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:27.693776 10026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:27.693965 10024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:27.693805 10021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:27.694072 10015 server_base.cc:1034] running on GCE node
I20260810 18:10:27.694674 10015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:27.695362 10015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:27.696540 10015 hybrid_clock.cc:648] HybridClock initialized: now 1786385427696500 us; error 51 us; skew 500 ppm
I20260810 18:10:27.698743 10015 webserver.cc:460] Webserver started at http://127.21.182.67:37841/ using document root <none> and password file <none>
I20260810 18:10:27.699421 10015 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:27.699492 10015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:27.699793 10015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:27.702230 10015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "03d6efda84154efaa4649aba2aeebde8"
format_stamp: "Formatted at 2026-08-10 18:10:27 on dist-test-slave-27sr"
server_key: "75a361202c00507cdad0d3a71cddc940"
server_key_iv: "95b92e87c26d553ff97d3b21f484fc49"
server_key_version: "encryptionkey@0"
I20260810 18:10:27.702853 10015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "03d6efda84154efaa4649aba2aeebde8"
format_stamp: "Formatted at 2026-08-10 18:10:27 on dist-test-slave-27sr"
server_key: "75a361202c00507cdad0d3a71cddc940"
server_key_iv: "95b92e87c26d553ff97d3b21f484fc49"
server_key_version: "encryptionkey@0"
I20260810 18:10:27.707032 10015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:10:27.709719 10033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:27.711217 10015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:10:27.711376 10015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "03d6efda84154efaa4649aba2aeebde8"
format_stamp: "Formatted at 2026-08-10 18:10:27 on dist-test-slave-27sr"
server_key: "75a361202c00507cdad0d3a71cddc940"
server_key_iv: "95b92e87c26d553ff97d3b21f484fc49"
server_key_version: "encryptionkey@0"
I20260810 18:10:27.711495 10015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:27.723981 10015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:27.724745 10015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:27.724972 10015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:27.725621 10015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:27.726660 10015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:27.726720 10015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:27.726797 10015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:27.726840 10015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:27.736886 10015 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:34417
I20260810 18:10:27.736913 10160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:34417 every 8 connection(s)
I20260810 18:10:27.737902 10015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:10:27.747058 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 10015
I20260810 18:10:27.747223 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:10:27.747599 22233 external_mini_cluster.cc:1428] Setting key 5f894b0a062a7a56f0faf98d36f7e36a
I20260810 18:10:27.748806 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.740140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:47759 (local address 127.21.182.126:36767)
0810 18:10:27.740324 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:27.740329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:27.741611 (+ 1282us) server_negotiation.cc:407] Connection header received
0810 18:10:27.743298 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:27.743301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:27.743352 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:27.743437 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:27.744475 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.744998 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.746282 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.746498 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.746579 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.748162 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:27.748169 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:27.748175 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:27.748194 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:27.748220 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:27.748234 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:27.748320 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:27.748560 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 18:10:27.748631 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:10:27.749912 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.740585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:36767 (local address 127.21.182.67:47759)
0810 18:10:27.741423 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:27.741485 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:27.743073 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:27.743581 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:27.743589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:27.743684 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:27.744336 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.744351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.745118 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.745121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.746161 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.746171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.746708 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.746715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.746995 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.747758 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:10:27.747761 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:27.747823 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:10:27.747956 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:27.748352 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:27.748364 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:27.748372 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 18:10:27.748514 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 18:10:27.748852 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":105,"threads_started":1}
I20260810 18:10:27.751098 10161 heartbeater.cc:347] Connected to a master server at 127.21.182.126:36767
I20260810 18:10:27.751320 10161 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:27.751817 10161 heartbeater.cc:511] Master 127.21.182.126:36767 requested a full tablet report, sending...
I20260810 18:10:27.752857 9631 ts_manager.cc:194] Registered new tserver with Master: 03d6efda84154efaa4649aba2aeebde8 (127.21.182.67:34417)
I20260810 18:10:27.753415 9631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.182.67:47759
I20260810 18:10:27.762173 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:10:27.768024 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.764499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37152 (local address 127.21.182.126:36767)
0810 18:10:27.764637 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:27.764641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:27.764750 (+ 109us) server_negotiation.cc:407] Connection header received
0810 18:10:27.764860 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:27.764862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:27.764907 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:27.764981 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:27.765246 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.765755 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.766558 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.766766 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.766844 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.767609 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:27.767613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:27.767615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:27.767627 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:27.767646 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:27.767654 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:27.767713 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:27.767791 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 18:10:27.767838 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:10:27.770844 9631 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:10:27.774076 9631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:10:27.789142 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.783008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34417 (local address 127.0.0.1:59640)
0810 18:10:27.783663 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:27.783677 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:27.783773 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:27.784854 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:27.784857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:27.784877 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:27.785013 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.785024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.786220 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.786223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.787358 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.787370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.788524 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.788530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.788680 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.788721 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 18:10:27.788844 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 18:10:27.788969 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":1,"thread_start_us":120,"threads_started":1}
I20260810 18:10:27.789269 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.782723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57824 (local address 127.21.182.65:34185)
0810 18:10:27.783299 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:27.783313 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:27.783342 (+ 29us) server_negotiation.cc:407] Connection header received
0810 18:10:27.783428 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:27.783435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:27.783711 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:27.784343 (+ 632us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:27.784490 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.785709 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.787639 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.788417 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.788639 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.788821 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:27.789013 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 18:10:27.789117 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":162,"threads_started":1}
I20260810 18:10:27.789217 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.782582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:34185 (local address 127.0.0.1:57824)
0810 18:10:27.783069 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:27.783117 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:27.783255 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:27.783945 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:27.783954 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:27.783983 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:27.784146 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.784165 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.785871 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.785874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.787501 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.787513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.788557 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.788561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.788841 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.788875 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 18:10:27.788977 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 18:10:27.789052 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":174,"thread_start_us":75,"threads_started":1}
I20260810 18:10:27.789577 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.783279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:43461 (local address 127.0.0.1:35966)
0810 18:10:27.784193 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:27.784208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:27.784305 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:27.784933 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:27.784936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:27.784965 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:27.785055 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.785061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.786511 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.786514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.787738 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.787745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.788864 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.788868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.788944 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.788963 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:10:27.789029 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 18:10:27.789087 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":32,"thread_start_us":115,"threads_started":1}
I20260810 18:10:27.789853 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.783462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.21.182.67:34417)
0810 18:10:27.784323 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:27.784339 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:27.784364 (+ 25us) server_negotiation.cc:407] Connection header received
0810 18:10:27.784459 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:27.784464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:27.784680 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:27.784876 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:27.785145 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.786025 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.787512 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.788374 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.788508 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.788723 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:27.789571 (+ 848us) server_negotiation.cc:299] Negotiation successful
0810 18:10:27.789687 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":191,"threads_started":1}
I20260810 18:10:27.789981 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.783608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.21.182.66:43461)
0810 18:10:27.784075 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:27.784089 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:27.784402 (+ 313us) server_negotiation.cc:407] Connection header received
0810 18:10:27.784492 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:27.784497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:27.784709 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:27.785180 (+ 471us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:27.785353 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.786370 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.787868 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.788472 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.788586 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.788799 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:27.789216 (+ 417us) server_negotiation.cc:299] Negotiation successful
0810 18:10:27.789819 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":157,"threads_started":1}
I20260810 18:10:27.792742 9781 tablet_service.cc:1467] Processing CreateTablet for tablet 2479929224694551b1a0b5733d15edf6 (DEFAULT_TABLE table=test-table [id=4e7a306184d64c84889c4b0fa9fb47d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:27.792753 10085 tablet_service.cc:1467] Processing CreateTablet for tablet 2479929224694551b1a0b5733d15edf6 (DEFAULT_TABLE table=test-table [id=4e7a306184d64c84889c4b0fa9fb47d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:27.792768 9930 tablet_service.cc:1467] Processing CreateTablet for tablet 2479929224694551b1a0b5733d15edf6 (DEFAULT_TABLE table=test-table [id=4e7a306184d64c84889c4b0fa9fb47d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:27.793793 9781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2479929224694551b1a0b5733d15edf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:27.793809 10085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2479929224694551b1a0b5733d15edf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:27.793809 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2479929224694551b1a0b5733d15edf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:27.800565 10180 tablet_bootstrap.cc:492] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01: Bootstrap starting.
I20260810 18:10:27.801505 10182 tablet_bootstrap.cc:492] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8: Bootstrap starting.
I20260810 18:10:27.802517 10181 tablet_bootstrap.cc:492] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97: Bootstrap starting.
I20260810 18:10:27.803174 10180 tablet_bootstrap.cc:654] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:27.803680 10182 tablet_bootstrap.cc:654] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:27.804177 10180 log.cc:824] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:27.804557 10182 log.cc:824] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:27.804648 10181 tablet_bootstrap.cc:654] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:27.805480 10181 log.cc:824] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:27.806442 10180 tablet_bootstrap.cc:492] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01: No bootstrap required, opened a new log
I20260810 18:10:27.806648 10180 ts_tablet_manager.cc:1397] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260810 18:10:27.806708 10182 tablet_bootstrap.cc:492] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8: No bootstrap required, opened a new log
I20260810 18:10:27.806874 10182 ts_tablet_manager.cc:1397] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 18:10:27.807261 10181 tablet_bootstrap.cc:492] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97: No bootstrap required, opened a new log
I20260810 18:10:27.807456 10181 ts_tablet_manager.cc:1397] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 18:10:27.810853 10181 raft_consensus.cc:348] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d519be7b2d435f9489e538d7705a01" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34185 } } peers { permanent_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43461 } } peers { permanent_uuid: "03d6efda84154efaa4649aba2aeebde8" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34417 } }
I20260810 18:10:27.811094 10181 raft_consensus.cc:374] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:27.811147 10181 raft_consensus.cc:729] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9b1baa75ded4b4c93e14bb1cb291d97, State: Initialized, Role: FOLLOWER
I20260810 18:10:27.811394 10180 raft_consensus.cc:348] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d519be7b2d435f9489e538d7705a01" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34185 } } peers { permanent_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43461 } } peers { permanent_uuid: "03d6efda84154efaa4649aba2aeebde8" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34417 } }
I20260810 18:10:27.811394 10182 raft_consensus.cc:348] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d519be7b2d435f9489e538d7705a01" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34185 } } peers { permanent_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43461 } } peers { permanent_uuid: "03d6efda84154efaa4649aba2aeebde8" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34417 } }
I20260810 18:10:27.811635 10182 raft_consensus.cc:374] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:27.811635 10180 raft_consensus.cc:374] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:27.811687 10182 raft_consensus.cc:729] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d6efda84154efaa4649aba2aeebde8, State: Initialized, Role: FOLLOWER
I20260810 18:10:27.811705 10180 raft_consensus.cc:729] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d519be7b2d435f9489e538d7705a01, State: Initialized, Role: FOLLOWER
I20260810 18:10:27.811631 10181 consensus_queue.cc:260] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d519be7b2d435f9489e538d7705a01" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34185 } } peers { permanent_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43461 } } peers { permanent_uuid: "03d6efda84154efaa4649aba2aeebde8" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34417 } }
I20260810 18:10:27.812148 10182 consensus_queue.cc:260] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d519be7b2d435f9489e538d7705a01" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34185 } } peers { permanent_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43461 } } peers { permanent_uuid: "03d6efda84154efaa4649aba2aeebde8" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34417 } }
I20260810 18:10:27.812148 10180 consensus_queue.cc:260] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d519be7b2d435f9489e538d7705a01" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34185 } } peers { permanent_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43461 } } peers { permanent_uuid: "03d6efda84154efaa4649aba2aeebde8" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34417 } }
I20260810 18:10:27.812776 10181 ts_tablet_manager.cc:1428] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260810 18:10:27.813148 10180 ts_tablet_manager.cc:1428] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260810 18:10:27.813160 10011 heartbeater.cc:503] Master 127.21.182.126:36767 was elected leader, sending a full tablet report...
I20260810 18:10:27.813148 10182 ts_tablet_manager.cc:1428] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:10:27.813479 9856 heartbeater.cc:503] Master 127.21.182.126:36767 was elected leader, sending a full tablet report...
I20260810 18:10:27.814467 10161 heartbeater.cc:503] Master 127.21.182.126:36767 was elected leader, sending a full tablet report...
W20260810 18:10:27.837641 9857 tablet.cc:2369] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:10:27.877199 10187 raft_consensus.cc:482] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:10:27.877434 10187 raft_consensus.cc:504] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d519be7b2d435f9489e538d7705a01" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34185 } } peers { permanent_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43461 } } peers { permanent_uuid: "03d6efda84154efaa4649aba2aeebde8" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34417 } }
I20260810 18:10:27.878731 10187 leader_election.cc:288] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d519be7b2d435f9489e538d7705a01 (127.21.182.65:34185), 03d6efda84154efaa4649aba2aeebde8 (127.21.182.67:34417)
I20260810 18:10:27.883236 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.879005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:34185 (local address 127.21.182.66:42283)
0810 18:10:27.879230 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:27.879247 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:27.879357 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:27.879709 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:27.879712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:27.879747 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:27.879970 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.879979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.881032 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.881036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.881947 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.881958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.882759 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.882764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.882912 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.882942 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 18:10:27.883004 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 18:10:27.883064 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":60}
I20260810 18:10:27.883322 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.879236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:42283 (local address 127.21.182.65:34185)
0810 18:10:27.879388 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:27.879394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:27.879418 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:10:27.879496 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:27.879503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:27.879573 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:27.879691 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:27.880131 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.880904 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.882107 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.882625 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.882731 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.882825 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:27.883058 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 18:10:27.883160 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 18:10:27.883394 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.879238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:55397 (local address 127.21.182.67:34417)
0810 18:10:27.879388 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:27.879394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:27.879484 (+ 90us) server_negotiation.cc:407] Connection header received
0810 18:10:27.879595 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:27.879601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:27.879663 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:27.879780 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:27.880515 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.881120 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.882287 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.882698 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.882819 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.882872 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:27.883180 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 18:10:27.883257 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:10:27.883854 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.879005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34417 (local address 127.21.182.66:55397)
0810 18:10:27.879386 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:27.879399 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:27.879500 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:27.879782 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:27.879784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:27.879798 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:27.879977 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.879982 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.881226 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.881229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.882145 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:27.882157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.882849 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.882855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:27.883024 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.883053 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 18:10:27.883138 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 18:10:27.883205 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":99,"thread_start_us":119,"threads_started":1}
I20260810 18:10:27.884091 9805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2479929224694551b1a0b5733d15edf6" candidate_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d519be7b2d435f9489e538d7705a01" is_pre_election: true
I20260810 18:10:27.884542 9805 raft_consensus.cc:2454] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9b1baa75ded4b4c93e14bb1cb291d97 in term 0.
I20260810 18:10:27.884557 10111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2479929224694551b1a0b5733d15edf6" candidate_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d6efda84154efaa4649aba2aeebde8" is_pre_election: true
I20260810 18:10:27.884850 10111 raft_consensus.cc:2454] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9b1baa75ded4b4c93e14bb1cb291d97 in term 0.
I20260810 18:10:27.885212 9878 leader_election.cc:302] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d519be7b2d435f9489e538d7705a01, c9b1baa75ded4b4c93e14bb1cb291d97; no voters:
I20260810 18:10:27.885607 10187 raft_consensus.cc:2781] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:10:27.885737 10187 raft_consensus.cc:482] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:10:27.885805 10187 raft_consensus.cc:3037] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:27.886961 10187 raft_consensus.cc:504] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d519be7b2d435f9489e538d7705a01" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34185 } } peers { permanent_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43461 } } peers { permanent_uuid: "03d6efda84154efaa4649aba2aeebde8" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34417 } }
I20260810 18:10:27.887403 10187 leader_election.cc:288] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [CANDIDATE]: Term 1 election: Requested vote from peers 14d519be7b2d435f9489e538d7705a01 (127.21.182.65:34185), 03d6efda84154efaa4649aba2aeebde8 (127.21.182.67:34417)
I20260810 18:10:27.887816 9805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2479929224694551b1a0b5733d15edf6" candidate_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d519be7b2d435f9489e538d7705a01"
I20260810 18:10:27.887864 10111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2479929224694551b1a0b5733d15edf6" candidate_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d6efda84154efaa4649aba2aeebde8"
I20260810 18:10:27.888010 9805 raft_consensus.cc:3037] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:27.888051 10111 raft_consensus.cc:3037] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:27.889168 10111 raft_consensus.cc:2454] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9b1baa75ded4b4c93e14bb1cb291d97 in term 1.
I20260810 18:10:27.889200 9805 raft_consensus.cc:2454] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9b1baa75ded4b4c93e14bb1cb291d97 in term 1.
I20260810 18:10:27.889596 9878 leader_election.cc:302] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03d6efda84154efaa4649aba2aeebde8, 14d519be7b2d435f9489e538d7705a01, c9b1baa75ded4b4c93e14bb1cb291d97; no voters:
I20260810 18:10:27.889786 10187 raft_consensus.cc:2781] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:27.890089 10187 raft_consensus.cc:686] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [term 1 LEADER]: Becoming Leader. State: Replica: c9b1baa75ded4b4c93e14bb1cb291d97, State: Running, Role: LEADER
I20260810 18:10:27.890465 10187 consensus_queue.cc:237] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d519be7b2d435f9489e538d7705a01" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34185 } } peers { permanent_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43461 } } peers { permanent_uuid: "03d6efda84154efaa4649aba2aeebde8" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34417 } }
I20260810 18:10:27.894254 9640 catalog_manager.cc:5495] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9b1baa75ded4b4c93e14bb1cb291d97 (127.21.182.66). New cstate: current_term: 1 leader_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d519be7b2d435f9489e538d7705a01" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9b1baa75ded4b4c93e14bb1cb291d97" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03d6efda84154efaa4649aba2aeebde8" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34417 } health_report { overall_health: UNKNOWN } } }
I20260810 18:10:27.913954 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.910084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.21.182.66:43461)
0810 18:10:27.910261 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:27.910268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:27.910286 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:10:27.910377 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:27.910379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:27.910435 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:27.910548 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:10:27.910950 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.911459 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.912377 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:27.912598 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:27.912713 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:27.913049 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:10:27.913511 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:10:27.913650 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:27.913732 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 18:10:27.913784 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:10:27.920604 10111 raft_consensus.cc:1261] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8 [term 1 FOLLOWER]: Refusing update from remote peer c9b1baa75ded4b4c93e14bb1cb291d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:10:27.920603 9805 raft_consensus.cc:1261] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01 [term 1 FOLLOWER]: Refusing update from remote peer c9b1baa75ded4b4c93e14bb1cb291d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:10:27.921438 10193 consensus_queue.cc:1035] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d6efda84154efaa4649aba2aeebde8" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:10:27.921624 10187 consensus_queue.cc:1035] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d519be7b2d435f9489e538d7705a01" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 34185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:10:27.939739 9638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:10:27.939988 9638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:10:27.942576 9638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 18b01d206d384180be33b610a68b9ada: Sending DeleteTablet for 3 replicas of tablet 2479929224694551b1a0b5733d15edf6
I20260810 18:10:27.943558 9781 tablet_service.cc:1514] Processing DeleteTablet for tablet 2479929224694551b1a0b5733d15edf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:27 UTC) from {username='slave'} at 127.0.0.1:57824
I20260810 18:10:27.943558 10085 tablet_service.cc:1514] Processing DeleteTablet for tablet 2479929224694551b1a0b5733d15edf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:27 UTC) from {username='slave'} at 127.0.0.1:59640
I20260810 18:10:27.944178 10207 tablet_replica.cc:330] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8: stopping tablet replica
I20260810 18:10:27.944260 9930 tablet_service.cc:1514] Processing DeleteTablet for tablet 2479929224694551b1a0b5733d15edf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:27 UTC) from {username='slave'} at 127.0.0.1:35966
I20260810 18:10:27.944336 10209 tablet_replica.cc:330] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01: stopping tablet replica
I20260810 18:10:27.944537 10207 raft_consensus.cc:2229] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:10:27.944583 10209 raft_consensus.cc:2229] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:10:27.944895 10207 raft_consensus.cc:2258] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:27.944895 10209 raft_consensus.cc:2258] T 2479929224694551b1a0b5733d15edf6 P 14d519be7b2d435f9489e538d7705a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:27.945279 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 9711
I20260810 18:10:27.947862 10207 ts_tablet_manager.cc:1905] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:10:27.951390 10207 ts_tablet_manager.cc:1918] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:10:27.951499 10207 log.cc:1196] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/2479929224694551b1a0b5733d15edf6
I20260810 18:10:27.951853 10207 ts_tablet_manager.cc:1939] T 2479929224694551b1a0b5733d15edf6 P 03d6efda84154efaa4649aba2aeebde8: Deleting consensus metadata
I20260810 18:10:27.953486 9622 catalog_manager.cc:4841] TS 03d6efda84154efaa4649aba2aeebde8 (127.21.182.67:34417): tablet 2479929224694551b1a0b5733d15edf6 (table test-table [id=4e7a306184d64c84889c4b0fa9fb47d9]) successfully deleted
I20260810 18:10:27.953788 10210 tablet_replica.cc:330] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97: stopping tablet replica
I20260810 18:10:27.954075 10210 raft_consensus.cc:2229] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:10:27.954404 10210 raft_consensus.cc:2258] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:27.956672 10210 ts_tablet_manager.cc:1905] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:10:27.960736 10210 ts_tablet_manager.cc:1918] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:10:27.960886 10210 log.cc:1196] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/2479929224694551b1a0b5733d15edf6
I20260810 18:10:27.961287 10210 ts_tablet_manager.cc:1939] T 2479929224694551b1a0b5733d15edf6 P c9b1baa75ded4b4c93e14bb1cb291d97: Deleting consensus metadata
I20260810 18:10:27.963009 9624 catalog_manager.cc:4841] TS c9b1baa75ded4b4c93e14bb1cb291d97 (127.21.182.66:43461): tablet 2479929224694551b1a0b5733d15edf6 (table test-table [id=4e7a306184d64c84889c4b0fa9fb47d9]) successfully deleted
W20260810 18:10:27.967413 9624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:34185 (error 108)
I20260810 18:10:27.968603 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 9859
I20260810 18:10:27.968884 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:27.968471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:34185 (local address 127.0.0.1:57838)
0810 18:10:27.968659 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:27.968760 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260810 18:10:27.970737 9624 catalog_manager.cc:4584] TS 14d519be7b2d435f9489e538d7705a01 (127.21.182.65:34185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2479929224694551b1a0b5733d15edf6: Network error: Client connection negotiation failed: client connection to 127.21.182.65:34185: connect: Connection refused (error 111)
I20260810 18:10:27.978367 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 10015
I20260810 18:10:27.989548 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 9602
2026-08-10T18:10:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1171 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:28 dist-test-slave-27sr krb5kdc[10214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:28 dist-test-slave-27sr krb5kdc[10214](info): set up 2 sockets
Aug 10 18:10:28 dist-test-slave-27sr krb5kdc[10214](info): commencing operation
krb5kdc: starting...
W20260810 18:10:29.134609 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:29 dist-test-slave-27sr krb5kdc[10214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:10:29.275401 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:40033
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:40033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:29.396749 10229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:29.397119 10229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:29.397205 10229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:29.401535 10229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:29.401656 10229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:29.401691 10229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:29.401731 10229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:29.401768 10229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:29.407466 10229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:40033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:29.408789 10229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:29.409950 10229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:29.420020 10235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:29.420233 10229 server_base.cc:1034] running on GCE node
W20260810 18:10:29.420020 10237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:29.420068 10234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:29.421113 10229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:29.422500 10229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:29.423751 10229 hybrid_clock.cc:648] HybridClock initialized: now 1786385429423736 us; error 33 us; skew 500 ppm
Aug 10 18:10:29 dist-test-slave-27sr krb5kdc[10214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385429, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:29.427001 10229 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:10:29.428215 10229 webserver.cc:460] Webserver started at http://127.21.182.126:35953/ using document root <none> and password file <none>
I20260810 18:10:29.428920 10229 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:29.429028 10229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:29.429304 10229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:29.431721 10229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "de72b993cfd54bc5b6d107db81907483"
format_stamp: "Formatted at 2026-08-10 18:10:29 on dist-test-slave-27sr"
server_key: "a87ff1ecb724781fdd4ea854647b1084"
server_key_iv: "86dc29545432ebfe0d2ab2486bd50b84"
server_key_version: "encryptionkey@0"
I20260810 18:10:29.432489 10229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "de72b993cfd54bc5b6d107db81907483"
format_stamp: "Formatted at 2026-08-10 18:10:29 on dist-test-slave-27sr"
server_key: "a87ff1ecb724781fdd4ea854647b1084"
server_key_iv: "86dc29545432ebfe0d2ab2486bd50b84"
server_key_version: "encryptionkey@0"
I20260810 18:10:29.437194 10229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:10:29.440575 10244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:29.442195 10229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:10:29.442417 10229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "de72b993cfd54bc5b6d107db81907483"
format_stamp: "Formatted at 2026-08-10 18:10:29 on dist-test-slave-27sr"
server_key: "a87ff1ecb724781fdd4ea854647b1084"
server_key_iv: "86dc29545432ebfe0d2ab2486bd50b84"
server_key_version: "encryptionkey@0"
I20260810 18:10:29.442584 10229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:29.475642 10229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:29.479260 10229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:29.479530 10229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:29.488363 10229 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:40033
I20260810 18:10:29.488394 10303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:40033 every 8 connection(s)
I20260810 18:10:29.489537 10229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:29.492666 10304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:29.494477 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 10229
I20260810 18:10:29.494601 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:29.494966 22233 external_mini_cluster.cc:1428] Setting key 8255dbc69d0e5235f764827e4e513aae
I20260810 18:10:29.500486 10304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483: Bootstrap starting.
I20260810 18:10:29.503537 10304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:29.504773 10304 log.cc:824] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:10:29 dist-test-slave-27sr krb5kdc[10214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:10:29.507376 10304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483: No bootstrap required, opened a new log
I20260810 18:10:29.511260 10304 raft_consensus.cc:348] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de72b993cfd54bc5b6d107db81907483" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40033 } }
I20260810 18:10:29.511544 10304 raft_consensus.cc:374] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:29.511629 10304 raft_consensus.cc:729] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de72b993cfd54bc5b6d107db81907483, State: Initialized, Role: FOLLOWER
I20260810 18:10:29.512117 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:29.496796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.21.182.126:40033)
0810 18:10:29.497315 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:29.497337 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:29.497386 (+ 49us) server_negotiation.cc:407] Connection header received
0810 18:10:29.498317 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:29.498349 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:29.498853 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:29.499297 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:29.500228 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:29.501708 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:29.502662 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:29.503391 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:29.503588 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:29.506370 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:29.506380 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:29.506399 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:29.506434 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:29.508610 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:29.509179 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:29.509185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:29.509191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:29.509322 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:29.509668 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:29.509674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:29.509681 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:29.510160 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:29.510409 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:29.510747 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 18:10:29.511080 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":146,"threads_started":1}
I20260810 18:10:29.512205 10304 consensus_queue.cc:260] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de72b993cfd54bc5b6d107db81907483" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40033 } }
I20260810 18:10:29.512430 10304 raft_consensus.cc:388] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:29.512513 10304 raft_consensus.cc:482] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:29.512609 10304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:29.513792 10304 raft_consensus.cc:504] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de72b993cfd54bc5b6d107db81907483" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40033 } }
I20260810 18:10:29.514173 10304 leader_election.cc:302] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de72b993cfd54bc5b6d107db81907483; no voters:
I20260810 18:10:29.514565 10304 leader_election.cc:288] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:29.514679 10310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:29.514995 10310 raft_consensus.cc:686] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [term 1 LEADER]: Becoming Leader. State: Replica: de72b993cfd54bc5b6d107db81907483, State: Running, Role: LEADER
I20260810 18:10:29.515398 10310 consensus_queue.cc:237] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de72b993cfd54bc5b6d107db81907483" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40033 } }
I20260810 18:10:29.516037 10304 sys_catalog.cc:564] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:29.517280 10312 sys_catalog.cc:455] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de72b993cfd54bc5b6d107db81907483. Latest consensus state: current_term: 1 leader_uuid: "de72b993cfd54bc5b6d107db81907483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de72b993cfd54bc5b6d107db81907483" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40033 } } }
I20260810 18:10:29.517431 10312 sys_catalog.cc:458] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:29.517288 10311 sys_catalog.cc:455] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de72b993cfd54bc5b6d107db81907483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de72b993cfd54bc5b6d107db81907483" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40033 } } }
I20260810 18:10:29.517506 10311 sys_catalog.cc:458] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:29.517829 10319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:29.521410 10319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:29.527995 10319 catalog_manager.cc:1347] Generated new cluster ID: 499a505ecbe74cdcb069a641ac52147e
I20260810 18:10:29.528100 10319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:29.544762 10319 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:29.545881 10319 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:29.559067 10319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483: Generated new TSK 0
I20260810 18:10:29.559947 10319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:10:29.600520 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40033
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:29.745338 10331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:29.745687 10331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:29.745733 10331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:29.751397 10331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:29.751482 10331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:29.751579 10331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:10:29.757495 10331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:29.758857 10331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:29.760138 10331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:29.771611 10337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:29.771611 10340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:29.771688 10342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:29.771668 10331 server_base.cc:1034] running on GCE node
I20260810 18:10:29.772493 10331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:29.773240 10331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:29.774425 10331 hybrid_clock.cc:648] HybridClock initialized: now 1786385429774401 us; error 34 us; skew 500 ppm
Aug 10 18:10:29 dist-test-slave-27sr krb5kdc[10214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385429, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:29.777842 10331 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:10:29.778987 10331 webserver.cc:460] Webserver started at http://127.21.182.65:41333/ using document root <none> and password file <none>
I20260810 18:10:29.779696 10331 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:29.779764 10331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:29.780062 10331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:29.782369 10331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "7c453e60d858418dbf0299da621b33d2"
format_stamp: "Formatted at 2026-08-10 18:10:29 on dist-test-slave-27sr"
server_key: "d0e373c645f43b220fe4c718fbf4b8a4"
server_key_iv: "ac231c6e1b60a2aae77a4d7df391ec20"
server_key_version: "encryptionkey@0"
I20260810 18:10:29.783032 10331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "7c453e60d858418dbf0299da621b33d2"
format_stamp: "Formatted at 2026-08-10 18:10:29 on dist-test-slave-27sr"
server_key: "d0e373c645f43b220fe4c718fbf4b8a4"
server_key_iv: "ac231c6e1b60a2aae77a4d7df391ec20"
server_key_version: "encryptionkey@0"
I20260810 18:10:29.787278 10331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:10:29.790251 10353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:29.791740 10331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:10:29.791911 10331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "7c453e60d858418dbf0299da621b33d2"
format_stamp: "Formatted at 2026-08-10 18:10:29 on dist-test-slave-27sr"
server_key: "d0e373c645f43b220fe4c718fbf4b8a4"
server_key_iv: "ac231c6e1b60a2aae77a4d7df391ec20"
server_key_version: "encryptionkey@0"
I20260810 18:10:29.792075 10331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:29.813625 10331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:29.818010 10331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:29.818243 10331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:29.819023 10331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:29.820164 10331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:29.820218 10331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:29.820262 10331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:29.820317 10331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:29.830968 10331 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:43667
I20260810 18:10:29.831007 10481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:43667 every 8 connection(s)
I20260810 18:10:29.832114 10331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:10:29.840962 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 10331
I20260810 18:10:29.841163 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:10:29.841600 22233 external_mini_cluster.cc:1428] Setting key fac959ec6fde110825ceed32d1de928e
Aug 10 18:10:29 dist-test-slave-27sr krb5kdc[10214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385429, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:10:29.849032 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:29.834208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:54453 (local address 127.21.182.126:40033)
0810 18:10:29.834411 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:29.834417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:29.835426 (+ 1009us) server_negotiation.cc:407] Connection header received
0810 18:10:29.836990 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:29.836994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:29.837054 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:29.837138 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:29.838542 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:29.839093 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:29.840717 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:29.840950 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:29.841038 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:29.844413 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:29.844417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:29.844419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:29.844445 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:29.845964 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:29.846767 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:29.846770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:29.846771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:29.846861 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:29.847393 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:29.847400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:29.847405 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:29.847690 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:29.847801 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:29.848586 (+ 785us) server_negotiation.cc:299] Negotiation successful
0810 18:10:29.848756 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 18:10:29.850023 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:29.834670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40033 (local address 127.21.182.65:54453)
0810 18:10:29.835220 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:29.835278 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:29.836718 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:29.837338 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:29.837346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:29.837828 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:29.838387 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:29.838401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:29.839256 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:29.839259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:29.840520 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:29.840532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:29.841150 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:29.841156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:29.841401 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:29.842129 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:10:29.842148 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:29.844211 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:29.846145 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:29.846156 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:29.846180 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:29.846631 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:29.846996 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:29.847002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:29.847006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:29.847265 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:29.847841 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:29.847848 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:29.848285 (+ 437us) client_negotiation.cc:769] Sending connection context
0810 18:10:29.848481 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 18:10:29.848926 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":127,"threads_started":1}
I20260810 18:10:29.851665 10482 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40033
I20260810 18:10:29.852020 10482 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:29.852608 10482 heartbeater.cc:511] Master 127.21.182.126:40033 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
I20260810 18:10:29.854606 10265 ts_manager.cc:194] Registered new tserver with Master: 7c453e60d858418dbf0299da621b33d2 (127.21.182.65:43667)
I20260810 18:10:29.855836 10265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:54453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:10:29.874581 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40033
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:30.003499 10487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:30.003830 10487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:30.003881 10487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:30.008600 10487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:30.008690 10487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:30.008793 10487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:10:30.014528 10487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:30.015831 10487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:30.017047 10487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:30.028270 10496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:30.028268 10493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:30.028345 10487 server_base.cc:1034] running on GCE node
W20260810 18:10:30.028229 10492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:30.028955 10487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:30.029811 10487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:30.031009 10487 hybrid_clock.cc:648] HybridClock initialized: now 1786385430030969 us; error 53 us; skew 500 ppm
Aug 10 18:10:30 dist-test-slave-27sr krb5kdc[10214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385430, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:30.034523 10487 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:10:30.035674 10487 webserver.cc:460] Webserver started at http://127.21.182.66:46003/ using document root <none> and password file <none>
I20260810 18:10:30.036481 10487 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:30.036562 10487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:30.036846 10487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:30.039328 10487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "cf5a9ad03fd34de29ba050a30b9a5096"
format_stamp: "Formatted at 2026-08-10 18:10:30 on dist-test-slave-27sr"
server_key: "2c0db72aa8a65b8ffda8bdae50a0aefc"
server_key_iv: "d2638e42767ea225e8fef00eb76c98d4"
server_key_version: "encryptionkey@0"
I20260810 18:10:30.040076 10487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "cf5a9ad03fd34de29ba050a30b9a5096"
format_stamp: "Formatted at 2026-08-10 18:10:30 on dist-test-slave-27sr"
server_key: "2c0db72aa8a65b8ffda8bdae50a0aefc"
server_key_iv: "d2638e42767ea225e8fef00eb76c98d4"
server_key_version: "encryptionkey@0"
I20260810 18:10:30.044560 10487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:10:30.048092 10504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:30.049681 10487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:10:30.049885 10487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "cf5a9ad03fd34de29ba050a30b9a5096"
format_stamp: "Formatted at 2026-08-10 18:10:30 on dist-test-slave-27sr"
server_key: "2c0db72aa8a65b8ffda8bdae50a0aefc"
server_key_iv: "d2638e42767ea225e8fef00eb76c98d4"
server_key_version: "encryptionkey@0"
I20260810 18:10:30.050024 10487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:30.076648 10487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:30.080776 10487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:30.081024 10487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:30.081739 10487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:30.082715 10487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:30.082767 10487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:30.082844 10487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:30.082880 10487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:30.092986 10487 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:33095
I20260810 18:10:30.093006 10635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:33095 every 8 connection(s)
I20260810 18:10:30.094065 10487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:10:30.103832 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 10487
I20260810 18:10:30.104029 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:10:30.104377 22233 external_mini_cluster.cc:1428] Setting key 06279d00828c71a5d78297847a8a84d6
Aug 10 18:10:30 dist-test-slave-27sr krb5kdc[10214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385430, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:10:30.110608 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.096237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:46167 (local address 127.21.182.126:40033)
0810 18:10:30.096404 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:30.096408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:30.097420 (+ 1012us) server_negotiation.cc:407] Connection header received
0810 18:10:30.098889 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:30.098892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:30.098960 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:30.099069 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:30.100461 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.101021 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.102272 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.102517 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.102635 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.105938 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:30.105942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:30.105944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:30.105978 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:30.108082 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:30.108681 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:30.108686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:30.108688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:30.108832 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:30.109336 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:30.109340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:30.109342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:30.109566 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:30.109688 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:30.110284 (+ 596us) server_negotiation.cc:299] Negotiation successful
0810 18:10:30.110436 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:10:30.111541 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.096679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40033 (local address 127.21.182.66:46167)
0810 18:10:30.097240 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:30.097291 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:30.098626 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:30.099266 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:30.099274 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:30.099678 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:30.100292 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:30.100308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.101175 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.101179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:30.102120 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:30.102130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.102705 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.102710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:30.102900 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.103618 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:10:30.103648 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:30.105728 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:30.108222 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:30.108230 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:30.108249 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:30.108544 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:30.108978 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:30.108981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:30.108982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:30.109211 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:30.109685 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:30.109693 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:30.110009 (+ 316us) client_negotiation.cc:769] Sending connection context
0810 18:10:30.110226 (+ 217us) client_negotiation.cc:240] Negotiation successful
0810 18:10:30.110607 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":126,"threads_started":1}
I20260810 18:10:30.113106 10636 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40033
I20260810 18:10:30.113413 10636 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:30.113998 10636 heartbeater.cc:511] Master 127.21.182.126:40033 requested a full tablet report, sending...
I20260810 18:10:30.115296 10265 ts_manager.cc:194] Registered new tserver with Master: cf5a9ad03fd34de29ba050a30b9a5096 (127.21.182.66:33095)
I20260810 18:10:30.116252 10265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:46167
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:10:30.139518 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40033
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:30.262732 10643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:30.263098 10643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:30.263176 10643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:30.267668 10643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:30.267778 10643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:30.267944 10643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:10:30.273789 10643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:30.275074 10643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:30.276249 10643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:30.286999 10653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:30.287098 10643 server_base.cc:1034] running on GCE node
W20260810 18:10:30.286995 10650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:30.287299 10651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:30.287765 10643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:30.288584 10643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:30.289774 10643 hybrid_clock.cc:648] HybridClock initialized: now 1786385430289753 us; error 36 us; skew 500 ppm
Aug 10 18:10:30 dist-test-slave-27sr krb5kdc[10214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385430, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:30.292980 10643 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:10:30.294075 10643 webserver.cc:460] Webserver started at http://127.21.182.67:39855/ using document root <none> and password file <none>
I20260810 18:10:30.294764 10643 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:30.294837 10643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:30.295106 10643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:30.297710 10643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "0042434990d841b4ba85dd0c2d9ffac5"
format_stamp: "Formatted at 2026-08-10 18:10:30 on dist-test-slave-27sr"
server_key: "873b46455e5918ebe1e620825dc9191a"
server_key_iv: "9d092afab4cb5f144d449ea2d1573601"
server_key_version: "encryptionkey@0"
I20260810 18:10:30.298354 10643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "0042434990d841b4ba85dd0c2d9ffac5"
format_stamp: "Formatted at 2026-08-10 18:10:30 on dist-test-slave-27sr"
server_key: "873b46455e5918ebe1e620825dc9191a"
server_key_iv: "9d092afab4cb5f144d449ea2d1573601"
server_key_version: "encryptionkey@0"
I20260810 18:10:30.302862 10643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:10:30.305929 10665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:30.307374 10643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:10:30.307521 10643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "0042434990d841b4ba85dd0c2d9ffac5"
format_stamp: "Formatted at 2026-08-10 18:10:30 on dist-test-slave-27sr"
server_key: "873b46455e5918ebe1e620825dc9191a"
server_key_iv: "9d092afab4cb5f144d449ea2d1573601"
server_key_version: "encryptionkey@0"
I20260810 18:10:30.307653 10643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:30.324050 10643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:30.327809 10643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:30.328076 10643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:30.328730 10643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:30.329668 10643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:30.329721 10643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:30.329766 10643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:30.329782 10643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:30.340319 10643 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:34255
I20260810 18:10:30.340394 10800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:34255 every 8 connection(s)
I20260810 18:10:30.341396 10643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:10:30.348243 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 10643
I20260810 18:10:30.348424 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:10:30.348799 22233 external_mini_cluster.cc:1428] Setting key ad116c6f747332c1cbcc0aa877e33330
Aug 10 18:10:30 dist-test-slave-27sr krb5kdc[10214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385430, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:10:30.357391 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.343553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:44547 (local address 127.21.182.126:40033)
0810 18:10:30.343712 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:30.343716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:30.344751 (+ 1035us) server_negotiation.cc:407] Connection header received
0810 18:10:30.346215 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:30.346218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:30.346286 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:30.346417 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:30.347755 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.348579 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.350232 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.350476 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.350589 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.353615 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:30.353618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:30.353621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:30.353649 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:30.355186 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:30.355745 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:30.355748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:30.355749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:30.355832 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:30.356262 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:30.356264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:30.356266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:30.356468 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:30.356553 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:30.357142 (+ 589us) server_negotiation.cc:299] Negotiation successful
0810 18:10:30.357259 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:10:30.358372 10803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.344046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40033 (local address 127.21.182.67:44547)
0810 18:10:30.344574 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:30.344624 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:30.345951 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:30.346611 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:30.346619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:30.347021 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:30.347582 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:30.347606 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.348818 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.348822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:30.350080 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:30.350092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.350669 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.350677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:30.350879 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.351601 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:10:30.351620 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:30.353416 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:30.355320 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:30.355327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:30.355342 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:30.355649 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:30.355995 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:30.355998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:30.355999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:30.356163 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:30.356564 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:30.356570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:30.356907 (+ 337us) client_negotiation.cc:769] Sending connection context
0810 18:10:30.357084 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 18:10:30.357376 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":126,"threads_started":1}
I20260810 18:10:30.359732 10801 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40033
I20260810 18:10:30.360039 10801 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:30.360633 10801 heartbeater.cc:511] Master 127.21.182.126:40033 requested a full tablet report, sending...
I20260810 18:10:30.361919 10265 ts_manager.cc:194] Registered new tserver with Master: 0042434990d841b4ba85dd0c2d9ffac5 (127.21.182.67:34255)
I20260810 18:10:30.362653 10265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:44547
I20260810 18:10:30.363744 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:10:30.374138 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.366077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.21.182.126:40033)
0810 18:10:30.366228 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:30.366233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:30.366448 (+ 215us) server_negotiation.cc:407] Connection header received
0810 18:10:30.366582 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:30.366584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:30.366626 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:30.366700 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:30.367564 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.368139 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.368975 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.369179 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.369255 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.370850 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:30.370853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:30.370855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:30.370878 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:30.372202 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:30.372694 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:30.372697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:30.372698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:30.372773 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:30.373160 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:30.373162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:30.373163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:30.373431 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:30.373572 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:30.373858 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 18:10:30.373943 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:10:30.377590 10265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:10:30.381284 10265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:10:30.398290 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.391587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.21.182.65:43667)
0810 18:10:30.393184 (+ 1597us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:30.393194 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:30.393218 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:10:30.393293 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:30.393298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:30.393498 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:30.393671 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:30.393909 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.395005 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.396608 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.397451 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.397564 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.397731 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:30.398024 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 18:10:30.398133 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":103,"threads_started":1}
I20260810 18:10:30.398288 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.391415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:43667 (local address 127.0.0.1:46616)
0810 18:10:30.392317 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:30.392379 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:30.392531 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:30.393640 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:30.393643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:30.393665 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:30.393784 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:30.393791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.395127 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.395135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:30.396482 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:30.396492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.397589 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.397592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:30.397675 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.397841 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 18:10:30.397967 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 18:10:30.398105 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":74,"spinlock_wait_cycles":3840,"thread_start_us":88,"threads_started":1}
I20260810 18:10:30.398633 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.391791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.21.182.66:33095)
0810 18:10:30.392520 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:30.392533 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:30.392561 (+ 28us) server_negotiation.cc:407] Connection header received
0810 18:10:30.392860 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:30.392867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:30.393124 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:30.393289 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:30.393619 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.394840 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.396900 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.397746 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.397851 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.397998 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:30.398369 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 18:10:30.398474 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":154,"threads_started":1}
I20260810 18:10:30.398294 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.391618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:33095 (local address 127.0.0.1:57638)
0810 18:10:30.392317 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:30.392399 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:30.392531 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:30.393281 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:30.393289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:30.393315 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:30.393477 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:30.393484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.395016 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.395019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:30.396756 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:30.396769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.397889 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.397892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:30.397989 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.398011 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:10:30.398076 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 18:10:30.398124 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":121,"spinlock_wait_cycles":8448,"thread_start_us":122,"threads_started":1}
I20260810 18:10:30.399458 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.392509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:34255 (local address 127.0.0.1:45550)
0810 18:10:30.392884 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:30.392899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:30.392989 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:30.393744 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:30.393747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:30.393764 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:30.393851 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:30.393860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.394946 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.394952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:30.396756 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:30.396769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.399090 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.399094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:30.399190 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.399215 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:10:30.399267 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 18:10:30.399333 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":191,"thread_start_us":100,"threads_started":1}
I20260810 18:10:30.400271 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.392831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.21.182.67:34255)
0810 18:10:30.393279 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:30.393292 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:30.393314 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:10:30.393385 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:30.393390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:30.393599 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:30.393990 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:30.394100 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.394807 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.398065 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.398939 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.399745 (+ 806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.399936 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:30.400007 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 18:10:30.400119 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":105,"threads_started":1}
I20260810 18:10:30.402269 10560 tablet_service.cc:1467] Processing CreateTablet for tablet 387cbc36d14943978eed1bb8aa077da8 (DEFAULT_TABLE table=test-table [id=be018fd7260140a1983084d34bcaae44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:30.402266 10407 tablet_service.cc:1467] Processing CreateTablet for tablet 387cbc36d14943978eed1bb8aa077da8 (DEFAULT_TABLE table=test-table [id=be018fd7260140a1983084d34bcaae44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:30.402544 10725 tablet_service.cc:1467] Processing CreateTablet for tablet 387cbc36d14943978eed1bb8aa077da8 (DEFAULT_TABLE table=test-table [id=be018fd7260140a1983084d34bcaae44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:30.403455 10560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387cbc36d14943978eed1bb8aa077da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:30.403455 10407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387cbc36d14943978eed1bb8aa077da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:30.403532 10725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387cbc36d14943978eed1bb8aa077da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:30.410192 10820 tablet_bootstrap.cc:492] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5: Bootstrap starting.
I20260810 18:10:30.411674 10821 tablet_bootstrap.cc:492] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2: Bootstrap starting.
I20260810 18:10:30.411674 10822 tablet_bootstrap.cc:492] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096: Bootstrap starting.
I20260810 18:10:30.413076 10820 tablet_bootstrap.cc:654] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:30.414011 10822 tablet_bootstrap.cc:654] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:30.414012 10821 tablet_bootstrap.cc:654] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:30.414961 10820 log.cc:824] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:30.415005 10821 log.cc:824] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:30.415508 10822 log.cc:824] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:30.417208 10820 tablet_bootstrap.cc:492] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5: No bootstrap required, opened a new log
I20260810 18:10:30.417420 10821 tablet_bootstrap.cc:492] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2: No bootstrap required, opened a new log
I20260810 18:10:30.417452 10820 ts_tablet_manager.cc:1397] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:10:30.417658 10821 ts_tablet_manager.cc:1397] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:10:30.417686 10822 tablet_bootstrap.cc:492] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096: No bootstrap required, opened a new log
I20260810 18:10:30.417862 10822 ts_tablet_manager.cc:1397] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20260810 18:10:30.420668 10820 raft_consensus.cc:348] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c453e60d858418dbf0299da621b33d2" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 43667 } } peers { permanent_uuid: "cf5a9ad03fd34de29ba050a30b9a5096" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 33095 } } peers { permanent_uuid: "0042434990d841b4ba85dd0c2d9ffac5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34255 } }
I20260810 18:10:30.420938 10820 raft_consensus.cc:374] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:30.421013 10820 raft_consensus.cc:729] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0042434990d841b4ba85dd0c2d9ffac5, State: Initialized, Role: FOLLOWER
I20260810 18:10:30.421548 10820 consensus_queue.cc:260] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c453e60d858418dbf0299da621b33d2" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 43667 } } peers { permanent_uuid: "cf5a9ad03fd34de29ba050a30b9a5096" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 33095 } } peers { permanent_uuid: "0042434990d841b4ba85dd0c2d9ffac5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34255 } }
I20260810 18:10:30.421928 10822 raft_consensus.cc:348] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c453e60d858418dbf0299da621b33d2" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 43667 } } peers { permanent_uuid: "cf5a9ad03fd34de29ba050a30b9a5096" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 33095 } } peers { permanent_uuid: "0042434990d841b4ba85dd0c2d9ffac5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34255 } }
I20260810 18:10:30.422263 10822 raft_consensus.cc:374] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:30.422379 10822 raft_consensus.cc:729] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5a9ad03fd34de29ba050a30b9a5096, State: Initialized, Role: FOLLOWER
I20260810 18:10:30.422299 10821 raft_consensus.cc:348] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c453e60d858418dbf0299da621b33d2" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 43667 } } peers { permanent_uuid: "cf5a9ad03fd34de29ba050a30b9a5096" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 33095 } } peers { permanent_uuid: "0042434990d841b4ba85dd0c2d9ffac5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34255 } }
I20260810 18:10:30.422622 10821 raft_consensus.cc:374] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:30.422721 10821 raft_consensus.cc:729] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c453e60d858418dbf0299da621b33d2, State: Initialized, Role: FOLLOWER
I20260810 18:10:30.422757 10801 heartbeater.cc:503] Master 127.21.182.126:40033 was elected leader, sending a full tablet report...
I20260810 18:10:30.423092 10820 ts_tablet_manager.cc:1428] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:10:30.423282 10821 consensus_queue.cc:260] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c453e60d858418dbf0299da621b33d2" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 43667 } } peers { permanent_uuid: "cf5a9ad03fd34de29ba050a30b9a5096" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 33095 } } peers { permanent_uuid: "0042434990d841b4ba85dd0c2d9ffac5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34255 } }
I20260810 18:10:30.423583 10822 consensus_queue.cc:260] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c453e60d858418dbf0299da621b33d2" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 43667 } } peers { permanent_uuid: "cf5a9ad03fd34de29ba050a30b9a5096" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 33095 } } peers { permanent_uuid: "0042434990d841b4ba85dd0c2d9ffac5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34255 } }
I20260810 18:10:30.424054 10482 heartbeater.cc:503] Master 127.21.182.126:40033 was elected leader, sending a full tablet report...
I20260810 18:10:30.424393 10821 ts_tablet_manager.cc:1428] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:10:30.424578 10822 ts_tablet_manager.cc:1428] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260810 18:10:30.426088 10636 heartbeater.cc:503] Master 127.21.182.126:40033 was elected leader, sending a full tablet report...
I20260810 18:10:30.432693 10826 raft_consensus.cc:482] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:10:30.432914 10826 raft_consensus.cc:504] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c453e60d858418dbf0299da621b33d2" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 43667 } } peers { permanent_uuid: "cf5a9ad03fd34de29ba050a30b9a5096" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 33095 } } peers { permanent_uuid: "0042434990d841b4ba85dd0c2d9ffac5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34255 } }
I20260810 18:10:30.434063 10826 leader_election.cc:288] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c453e60d858418dbf0299da621b33d2 (127.21.182.65:43667), cf5a9ad03fd34de29ba050a30b9a5096 (127.21.182.66:33095)
I20260810 18:10:30.437793 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.434408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:35651 (local address 127.21.182.66:33095)
0810 18:10:30.434534 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:30.434539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:30.434553 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:10:30.434703 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:30.434706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:30.434757 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:30.434849 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:30.435124 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.435638 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.436810 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.437211 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.437349 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.437403 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:30.437590 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 18:10:30.437665 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:10:30.437798 10803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.434297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:33095 (local address 127.21.182.67:35651)
0810 18:10:30.434445 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:30.434461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:30.434565 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:30.434860 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:30.434862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:30.434893 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:30.435015 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:30.435021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.435757 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.435761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:30.436674 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:30.436684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.437342 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.437345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:30.437451 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.437473 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:10:30.437573 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 18:10:30.437657 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":42,"spinlock_wait_cycles":1280}
I20260810 18:10:30.438043 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.434390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:58435 (local address 127.21.182.65:43667)
0810 18:10:30.434536 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:30.434543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:30.434838 (+ 295us) server_negotiation.cc:407] Connection header received
0810 18:10:30.434991 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:30.434994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:30.435046 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:30.435130 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:30.435413 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.435965 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.437069 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.437481 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.437614 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.437666 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:30.437857 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 18:10:30.437939 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:10:30.438064 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.434299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:43667 (local address 127.21.182.67:58435)
0810 18:10:30.434741 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:30.434753 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:30.434879 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:30.435141 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:30.435144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:30.435159 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:30.435294 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:30.435299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.436083 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.436086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:30.436937 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:30.436947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.437606 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.437610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:30.437706 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.437744 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 18:10:30.437839 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 18:10:30.437911 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":114,"thread_start_us":140,"threads_started":1}
I20260810 18:10:30.438629 10583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387cbc36d14943978eed1bb8aa077da8" candidate_uuid: "0042434990d841b4ba85dd0c2d9ffac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf5a9ad03fd34de29ba050a30b9a5096" is_pre_election: true
I20260810 18:10:30.438715 10430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387cbc36d14943978eed1bb8aa077da8" candidate_uuid: "0042434990d841b4ba85dd0c2d9ffac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c453e60d858418dbf0299da621b33d2" is_pre_election: true
I20260810 18:10:30.439002 10583 raft_consensus.cc:2454] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0042434990d841b4ba85dd0c2d9ffac5 in term 0.
I20260810 18:10:30.439002 10430 raft_consensus.cc:2454] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0042434990d841b4ba85dd0c2d9ffac5 in term 0.
I20260810 18:10:30.439644 10667 leader_election.cc:302] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0042434990d841b4ba85dd0c2d9ffac5, 7c453e60d858418dbf0299da621b33d2, cf5a9ad03fd34de29ba050a30b9a5096; no voters:
I20260810 18:10:30.440016 10826 raft_consensus.cc:2781] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:10:30.440129 10826 raft_consensus.cc:482] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:10:30.440196 10826 raft_consensus.cc:3037] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:30.441593 10826 raft_consensus.cc:504] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c453e60d858418dbf0299da621b33d2" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 43667 } } peers { permanent_uuid: "cf5a9ad03fd34de29ba050a30b9a5096" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 33095 } } peers { permanent_uuid: "0042434990d841b4ba85dd0c2d9ffac5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34255 } }
I20260810 18:10:30.441915 10826 leader_election.cc:288] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [CANDIDATE]: Term 1 election: Requested vote from peers 7c453e60d858418dbf0299da621b33d2 (127.21.182.65:43667), cf5a9ad03fd34de29ba050a30b9a5096 (127.21.182.66:33095)
I20260810 18:10:30.442361 10430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387cbc36d14943978eed1bb8aa077da8" candidate_uuid: "0042434990d841b4ba85dd0c2d9ffac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c453e60d858418dbf0299da621b33d2"
I20260810 18:10:30.442358 10583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387cbc36d14943978eed1bb8aa077da8" candidate_uuid: "0042434990d841b4ba85dd0c2d9ffac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf5a9ad03fd34de29ba050a30b9a5096"
I20260810 18:10:30.442505 10430 raft_consensus.cc:3037] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:30.442505 10583 raft_consensus.cc:3037] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:30.443929 10583 raft_consensus.cc:2454] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0042434990d841b4ba85dd0c2d9ffac5 in term 1.
I20260810 18:10:30.443922 10430 raft_consensus.cc:2454] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0042434990d841b4ba85dd0c2d9ffac5 in term 1.
I20260810 18:10:30.444317 10667 leader_election.cc:302] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0042434990d841b4ba85dd0c2d9ffac5, 7c453e60d858418dbf0299da621b33d2, cf5a9ad03fd34de29ba050a30b9a5096; no voters:
I20260810 18:10:30.444478 10826 raft_consensus.cc:2781] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:30.444702 10826 raft_consensus.cc:686] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [term 1 LEADER]: Becoming Leader. State: Replica: 0042434990d841b4ba85dd0c2d9ffac5, State: Running, Role: LEADER
I20260810 18:10:30.445068 10826 consensus_queue.cc:237] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c453e60d858418dbf0299da621b33d2" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 43667 } } peers { permanent_uuid: "cf5a9ad03fd34de29ba050a30b9a5096" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 33095 } } peers { permanent_uuid: "0042434990d841b4ba85dd0c2d9ffac5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34255 } }
I20260810 18:10:30.448390 10265 catalog_manager.cc:5495] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0042434990d841b4ba85dd0c2d9ffac5 (127.21.182.67). New cstate: current_term: 1 leader_uuid: "0042434990d841b4ba85dd0c2d9ffac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c453e60d858418dbf0299da621b33d2" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf5a9ad03fd34de29ba050a30b9a5096" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 33095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0042434990d841b4ba85dd0c2d9ffac5" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 34255 } health_report { overall_health: HEALTHY } } }
I20260810 18:10:30.462512 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.459193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.21.182.67:34255)
0810 18:10:30.459324 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:30.459327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:30.459342 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:10:30.459409 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:30.459411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:30.459462 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:30.459559 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:10:30.459780 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.460351 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.461196 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.461406 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.461514 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.461757 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:10:30.462160 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:10:30.462267 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:30.462366 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 18:10:30.462401 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:10:30.468881 10583 raft_consensus.cc:1261] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096 [term 1 FOLLOWER]: Refusing update from remote peer 0042434990d841b4ba85dd0c2d9ffac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:10:30.468881 10430 raft_consensus.cc:1261] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2 [term 1 FOLLOWER]: Refusing update from remote peer 0042434990d841b4ba85dd0c2d9ffac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:10:30.469703 10830 consensus_queue.cc:1035] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c453e60d858418dbf0299da621b33d2" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:10:30.469929 10826 consensus_queue.cc:1035] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf5a9ad03fd34de29ba050a30b9a5096" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 33095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:10:30.481603 10835 mvcc.cc:205] Tried to move back new op lower bound from 7317034723192549376 to 7317034723105914880. Current Snapshot: MvccSnapshot[applied={T|T < 7317034723192549376}]
I20260810 18:10:30.481608 10834 mvcc.cc:205] Tried to move back new op lower bound from 7317034723192549376 to 7317034723105914880. Current Snapshot: MvccSnapshot[applied={T|T < 7317034723192549376}]
I20260810 18:10:30.486586 10265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:10:30.486778 10265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:10:30.489393 10265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de72b993cfd54bc5b6d107db81907483: Sending DeleteTablet for 3 replicas of tablet 387cbc36d14943978eed1bb8aa077da8
I20260810 18:10:30.490160 10560 tablet_service.cc:1514] Processing DeleteTablet for tablet 387cbc36d14943978eed1bb8aa077da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:30 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:57638
I20260810 18:10:30.490200 10407 tablet_service.cc:1514] Processing DeleteTablet for tablet 387cbc36d14943978eed1bb8aa077da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:30 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:46616
I20260810 18:10:30.490865 10844 tablet_replica.cc:330] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2: stopping tablet replica
I20260810 18:10:30.491029 10725 tablet_service.cc:1514] Processing DeleteTablet for tablet 387cbc36d14943978eed1bb8aa077da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:30 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:45550
I20260810 18:10:30.491251 10844 raft_consensus.cc:2229] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:10:30.491670 10844 raft_consensus.cc:2258] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:30.492271 10843 tablet_replica.cc:330] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096: stopping tablet replica
I20260810 18:10:30.492674 10843 raft_consensus.cc:2229] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:10:30.492988 10843 raft_consensus.cc:2258] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:30.493604 10844 ts_tablet_manager.cc:1905] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:10:30.493768 10845 tablet_replica.cc:330] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5: stopping tablet replica
I20260810 18:10:30.494107 10845 raft_consensus.cc:2229] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:10:30.494304 10843 ts_tablet_manager.cc:1905] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:10:30.494623 10845 raft_consensus.cc:2258] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:30.496872 10844 ts_tablet_manager.cc:1918] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:10:30.497071 10844 log.cc:1196] T 387cbc36d14943978eed1bb8aa077da8 P 7c453e60d858418dbf0299da621b33d2: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-0/wal/wals/387cbc36d14943978eed1bb8aa077da8
I20260810 18:10:30.497143 10843 ts_tablet_manager.cc:1918] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:10:30.497274 10843 log.cc:1196] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/387cbc36d14943978eed1bb8aa077da8
I20260810 18:10:30.497689 10843 ts_tablet_manager.cc:1939] T 387cbc36d14943978eed1bb8aa077da8 P cf5a9ad03fd34de29ba050a30b9a5096: Deleting consensus metadata
I20260810 18:10:30.497700 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 10331
I20260810 18:10:30.499109 10247 catalog_manager.cc:4841] TS cf5a9ad03fd34de29ba050a30b9a5096 (127.21.182.66:33095): tablet 387cbc36d14943978eed1bb8aa077da8 (table test-table [id=be018fd7260140a1983084d34bcaae44]) successfully deleted
I20260810 18:10:30.507591 10845 ts_tablet_manager.cc:1905] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:10:30.510774 10845 ts_tablet_manager.cc:1918] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:10:30.510984 10845 log.cc:1196] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/387cbc36d14943978eed1bb8aa077da8
I20260810 18:10:30.511399 10845 ts_tablet_manager.cc:1939] T 387cbc36d14943978eed1bb8aa077da8 P 0042434990d841b4ba85dd0c2d9ffac5: Deleting consensus metadata
I20260810 18:10:30.512977 10249 catalog_manager.cc:4841] TS 0042434990d841b4ba85dd0c2d9ffac5 (127.21.182.67:34255): tablet 387cbc36d14943978eed1bb8aa077da8 (table test-table [id=be018fd7260140a1983084d34bcaae44]) successfully deleted
W20260810 18:10:30.521219 10246 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:43667 (error 108)
I20260810 18:10:30.522292 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 10487
I20260810 18:10:30.522393 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.522059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:43667 (local address 127.0.0.1:46632)
0810 18:10:30.522217 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:30.522291 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 18:10:30.523720 10246 catalog_manager.cc:4584] TS 7c453e60d858418dbf0299da621b33d2 (127.21.182.65:43667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 387cbc36d14943978eed1bb8aa077da8: Network error: Client connection negotiation failed: client connection to 127.21.182.65:43667: connect: Connection refused (error 111)
I20260810 18:10:30.535225 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 10643
I20260810 18:10:30.545698 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 10229
2026-08-10T18:10:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2554 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T18:10:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:30Z Disabled control of system clock
I20260810 18:10:30.581123 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:40581
--webserver_interface=127.21.182.126
--webserver_port=0
--builtin_ntp_servers=127.21.182.84:34865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:40581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:10:30.702025 10850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:30.702361 10850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:30.702407 10850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:30.706589 10850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:30.706673 10850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:30.706688 10850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:30.706707 10850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:30.706722 10850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:30.712070 10850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:34865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:40581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:40581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:30.713421 10850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:30.715034 10850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:30.724831 10857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:30.724819 10859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:30.724819 10856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:30.725212 10850 server_base.cc:1034] running on GCE node
I20260810 18:10:30.725968 10850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:30.727458 10850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:30.728710 10850 hybrid_clock.cc:648] HybridClock initialized: now 1786385430728673 us; error 57 us; skew 500 ppm
I20260810 18:10:30.731089 10850 webserver.cc:460] Webserver started at http://127.21.182.126:42951/ using document root <none> and password file <none>
I20260810 18:10:30.731765 10850 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:30.731841 10850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:30.732139 10850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:30.734596 10850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "5b554b96694e4000ab1cf229f1a48e91"
format_stamp: "Formatted at 2026-08-10 18:10:30 on dist-test-slave-27sr"
server_key: "890d512034efe0a0e59034eec8cd36fb"
server_key_iv: "8706beb0a05822ee95d55be4b7ae9ca9"
server_key_version: "encryptionkey@0"
I20260810 18:10:30.735227 10850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "5b554b96694e4000ab1cf229f1a48e91"
format_stamp: "Formatted at 2026-08-10 18:10:30 on dist-test-slave-27sr"
server_key: "890d512034efe0a0e59034eec8cd36fb"
server_key_iv: "8706beb0a05822ee95d55be4b7ae9ca9"
server_key_version: "encryptionkey@0"
I20260810 18:10:30.740069 10850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 18:10:30.743144 10865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:30.744987 10850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:30.745231 10850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "5b554b96694e4000ab1cf229f1a48e91"
format_stamp: "Formatted at 2026-08-10 18:10:30 on dist-test-slave-27sr"
server_key: "890d512034efe0a0e59034eec8cd36fb"
server_key_iv: "8706beb0a05822ee95d55be4b7ae9ca9"
server_key_version: "encryptionkey@0"
I20260810 18:10:30.745393 10850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:30.766651 10850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:30.767583 10850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:30.767864 10850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:30.776191 10850 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:40581
I20260810 18:10:30.776214 10928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:40581 every 8 connection(s)
I20260810 18:10:30.777365 10850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:30.778349 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 10850
I20260810 18:10:30.778505 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:30.778854 22233 external_mini_cluster.cc:1428] Setting key a3277b0a1ec5ca8acfba1ec4e2e71cd1
I20260810 18:10:30.781639 10929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:30.787792 10929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91: Bootstrap starting.
I20260810 18:10:30.789968 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:30.780515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44812 (local address 127.21.182.126:40581)
0810 18:10:30.781281 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:30.781299 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:30.781349 (+ 50us) server_negotiation.cc:407] Connection header received
0810 18:10:30.782245 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:30.782268 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:30.782725 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:30.783139 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:30.783633 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.784862 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.785917 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:30.786483 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:30.786686 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:30.787936 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:30.787945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:30.787966 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:30.788421 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:30.788484 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:30.788507 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:30.788627 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:30.788801 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 18:10:30.789089 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":166,"threads_started":1}
I20260810 18:10:30.790514 10929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:30.791471 10929 log.cc:824] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:30.793797 10929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91: No bootstrap required, opened a new log
I20260810 18:10:30.797996 10929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b554b96694e4000ab1cf229f1a48e91" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40581 } }
I20260810 18:10:30.798278 10929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:30.798316 10929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b554b96694e4000ab1cf229f1a48e91, State: Initialized, Role: FOLLOWER
I20260810 18:10:30.798754 10929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b554b96694e4000ab1cf229f1a48e91" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40581 } }
I20260810 18:10:30.798878 10929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:30.798929 10929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:30.799005 10929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:30.800052 10929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b554b96694e4000ab1cf229f1a48e91" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40581 } }
I20260810 18:10:30.800395 10929 leader_election.cc:302] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b554b96694e4000ab1cf229f1a48e91; no voters:
I20260810 18:10:30.800711 10929 leader_election.cc:288] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:30.800833 10935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:30.801133 10935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [term 1 LEADER]: Becoming Leader. State: Replica: 5b554b96694e4000ab1cf229f1a48e91, State: Running, Role: LEADER
I20260810 18:10:30.801534 10935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b554b96694e4000ab1cf229f1a48e91" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40581 } }
I20260810 18:10:30.802098 10929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:30.804253 10936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b554b96694e4000ab1cf229f1a48e91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b554b96694e4000ab1cf229f1a48e91" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40581 } } }
I20260810 18:10:30.804404 10936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:30.804548 10937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b554b96694e4000ab1cf229f1a48e91. Latest consensus state: current_term: 1 leader_uuid: "5b554b96694e4000ab1cf229f1a48e91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b554b96694e4000ab1cf229f1a48e91" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 40581 } } }
I20260810 18:10:30.804693 10937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91 [sys.catalog]: This master's current role is: LEADER
W20260810 18:10:30.807762 10950 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:10:30.807868 10950 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:10:30.808001 10951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:30.808768 10951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:30.814994 10951 catalog_manager.cc:1347] Generated new cluster ID: d70c6a31adec4fc4a9996cadd849a965
I20260810 18:10:30.815106 10951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:30.842877 10951 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:30.844076 10951 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:30.852857 10951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91: Generated new TSK 0
I20260810 18:10:30.853700 10951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:30.859783 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40581
--builtin_ntp_servers=127.21.182.84:34865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:10:30.996618 10955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:30.996960 10955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:30.997036 10955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:31.002040 10955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:31.002121 10955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:31.002233 10955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:10:31.008414 10955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:34865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:31.009769 10955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:31.011376 10955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:31.022706 10965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:31.022914 10955 server_base.cc:1034] running on GCE node
W20260810 18:10:31.022742 10960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:31.022706 10961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:31.023500 10955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:31.024349 10955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:31.025568 10955 hybrid_clock.cc:648] HybridClock initialized: now 1786385431025550 us; error 29 us; skew 500 ppm
I20260810 18:10:31.027706 10955 webserver.cc:460] Webserver started at http://127.21.182.65:37099/ using document root <none> and password file <none>
I20260810 18:10:31.028455 10955 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:31.028582 10955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:31.028851 10955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:31.031213 10955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "eea1a9e285f74c8681e6230d6be4c205"
format_stamp: "Formatted at 2026-08-10 18:10:31 on dist-test-slave-27sr"
server_key: "edff9539103bfd1495119269a2506cae"
server_key_iv: "6d9b0c1e7c08a370dd37f2b9a07138d5"
server_key_version: "encryptionkey@0"
I20260810 18:10:31.031864 10955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "eea1a9e285f74c8681e6230d6be4c205"
format_stamp: "Formatted at 2026-08-10 18:10:31 on dist-test-slave-27sr"
server_key: "edff9539103bfd1495119269a2506cae"
server_key_iv: "6d9b0c1e7c08a370dd37f2b9a07138d5"
server_key_version: "encryptionkey@0"
I20260810 18:10:31.036305 10955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:10:31.039240 10971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:31.040616 10955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:31.040840 10955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "eea1a9e285f74c8681e6230d6be4c205"
format_stamp: "Formatted at 2026-08-10 18:10:31 on dist-test-slave-27sr"
server_key: "edff9539103bfd1495119269a2506cae"
server_key_iv: "6d9b0c1e7c08a370dd37f2b9a07138d5"
server_key_version: "encryptionkey@0"
I20260810 18:10:31.040983 10955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:31.070578 10955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:31.071395 10955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:31.071619 10955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:31.072363 10955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:31.073410 10955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:31.073485 10955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:31.073556 10955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:31.073607 10955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:31.082863 10955 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:35825
I20260810 18:10:31.082886 11106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:35825 every 8 connection(s)
I20260810 18:10:31.083999 10955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:10:31.089433 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 10955
I20260810 18:10:31.089660 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:10:31.090212 22233 external_mini_cluster.cc:1428] Setting key c7d5bf133a11d73ebf3bb843887a4684
I20260810 18:10:31.092715 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40581
--builtin_ntp_servers=127.21.182.84:34865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:10:31.096517 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.087080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:43043 (local address 127.21.182.126:40581)
0810 18:10:31.087275 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:31.087280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:31.088015 (+ 735us) server_negotiation.cc:407] Connection header received
0810 18:10:31.089552 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:31.089557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:31.089628 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:31.089712 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:31.090862 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.091575 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.093707 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.094102 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.094240 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.095772 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:31.095777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:31.095783 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:31.095801 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:31.095828 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:31.095841 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:31.095935 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:31.096211 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 18:10:31.096310 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 18:10:31.098351 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.086916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40581 (local address 127.21.182.65:43043)
0810 18:10:31.087837 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:31.087918 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:31.089308 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:31.089991 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:31.089998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:31.090094 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:31.090696 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.090712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.091744 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.091752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.093512 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.093527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.094307 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.094314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.094562 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.095375 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:10:31.095378 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:31.095444 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:10:31.095549 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:31.096016 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:31.096029 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:31.096038 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:10:31.096722 (+ 684us) client_negotiation.cc:240] Negotiation successful
0810 18:10:31.097125 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":153,"threads_started":1}
I20260810 18:10:31.100243 11107 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40581
I20260810 18:10:31.100548 11107 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:31.101222 11107 heartbeater.cc:511] Master 127.21.182.126:40581 requested a full tablet report, sending...
I20260810 18:10:31.103240 10887 ts_manager.cc:194] Registered new tserver with Master: eea1a9e285f74c8681e6230d6be4c205 (127.21.182.65:35825)
I20260810 18:10:31.105140 10887 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.182.65:43043
W20260810 18:10:31.219210 11110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:31.219514 11110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:31.219555 11110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:31.223799 11110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:31.223881 11110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:31.223964 11110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:10:31.229085 11110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:34865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:31.230475 11110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:31.232072 11110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:10:31.243224 11110 server_base.cc:1034] running on GCE node
W20260810 18:10:31.243175 11116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:31.243168 11115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:31.243175 11118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:31.243947 11110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:31.244788 11110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:31.245990 11110 hybrid_clock.cc:648] HybridClock initialized: now 1786385431245966 us; error 35 us; skew 500 ppm
I20260810 18:10:31.248298 11110 webserver.cc:460] Webserver started at http://127.21.182.66:45743/ using document root <none> and password file <none>
I20260810 18:10:31.249001 11110 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:31.249135 11110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:31.249532 11110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:31.252451 11110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "e30c35b03d7946eaa93184e82e6db917"
format_stamp: "Formatted at 2026-08-10 18:10:31 on dist-test-slave-27sr"
server_key: "30997566d22caa68fd33a685919379e9"
server_key_iv: "a2cab4bf36bc3f97a5616a89dee2bab0"
server_key_version: "encryptionkey@0"
I20260810 18:10:31.253136 11110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "e30c35b03d7946eaa93184e82e6db917"
format_stamp: "Formatted at 2026-08-10 18:10:31 on dist-test-slave-27sr"
server_key: "30997566d22caa68fd33a685919379e9"
server_key_iv: "a2cab4bf36bc3f97a5616a89dee2bab0"
server_key_version: "encryptionkey@0"
I20260810 18:10:31.257539 11110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:10:31.260459 11130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:31.261941 11110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:31.262128 11110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "e30c35b03d7946eaa93184e82e6db917"
format_stamp: "Formatted at 2026-08-10 18:10:31 on dist-test-slave-27sr"
server_key: "30997566d22caa68fd33a685919379e9"
server_key_iv: "a2cab4bf36bc3f97a5616a89dee2bab0"
server_key_version: "encryptionkey@0"
I20260810 18:10:31.262288 11110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:31.278626 11110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:31.279520 11110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:31.279776 11110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:31.280566 11110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:31.281813 11110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:31.281919 11110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:31.282034 11110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:31.282099 11110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:31.292888 11110 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:39709
I20260810 18:10:31.292912 11259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:39709 every 8 connection(s)
I20260810 18:10:31.293985 11110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:10:31.301827 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 11110
I20260810 18:10:31.301942 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:10:31.302309 22233 external_mini_cluster.cc:1428] Setting key 1ab35f4cf8068042d7198cafbbb953c3
I20260810 18:10:31.304844 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40581
--builtin_ntp_servers=127.21.182.84:34865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:10:31.305095 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.296275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:56979 (local address 127.21.182.126:40581)
0810 18:10:31.296448 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:31.296454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:31.297737 (+ 1283us) server_negotiation.cc:407] Connection header received
0810 18:10:31.299247 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:31.299251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:31.299312 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:31.299401 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:31.300481 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.301020 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.302315 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.302652 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.302781 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.304434 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:31.304438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:31.304441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:31.304457 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:31.304481 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:31.304492 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:31.304581 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:31.304798 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 18:10:31.304895 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:10:31.306030 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.296842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40581 (local address 127.21.182.66:56979)
0810 18:10:31.297566 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:31.297610 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:31.298971 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:31.299593 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:31.299600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:31.299692 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:31.300306 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.300320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.301152 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.301155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.302161 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.302172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.302861 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.302866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.303181 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.304058 (+ 877us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:10:31.304062 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:31.304125 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:10:31.304266 (+ 141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:31.304588 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:31.304596 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:31.304636 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 18:10:31.304773 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 18:10:31.305141 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":135,"threads_started":1}
I20260810 18:10:31.307421 11260 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40581
I20260810 18:10:31.307763 11260 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:31.308372 11260 heartbeater.cc:511] Master 127.21.182.126:40581 requested a full tablet report, sending...
I20260810 18:10:31.309563 10887 ts_manager.cc:194] Registered new tserver with Master: e30c35b03d7946eaa93184e82e6db917 (127.21.182.66:39709)
I20260810 18:10:31.310165 10887 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.182.66:56979
W20260810 18:10:31.430167 11263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:31.430495 11263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:31.430536 11263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:31.434737 11263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:31.434808 11263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:31.434870 11263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:10:31.440464 11263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:34865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:40581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:31.441630 11263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:31.443109 11263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:31.454107 11272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:31.454097 11269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:31.454097 11268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:31.454432 11263 server_base.cc:1034] running on GCE node
I20260810 18:10:31.455008 11263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:31.455721 11263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:31.456929 11263 hybrid_clock.cc:648] HybridClock initialized: now 1786385431456902 us; error 37 us; skew 500 ppm
I20260810 18:10:31.459116 11263 webserver.cc:460] Webserver started at http://127.21.182.67:33113/ using document root <none> and password file <none>
I20260810 18:10:31.459767 11263 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:31.459868 11263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:31.460127 11263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:31.462330 11263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "9a8f5609d62d44b9ae5b248754f25527"
format_stamp: "Formatted at 2026-08-10 18:10:31 on dist-test-slave-27sr"
server_key: "bf380a219a168f364fc15cc94152de0c"
server_key_iv: "fd572bf3c17ee53a7fcc696f795ab337"
server_key_version: "encryptionkey@0"
I20260810 18:10:31.462949 11263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "9a8f5609d62d44b9ae5b248754f25527"
format_stamp: "Formatted at 2026-08-10 18:10:31 on dist-test-slave-27sr"
server_key: "bf380a219a168f364fc15cc94152de0c"
server_key_iv: "fd572bf3c17ee53a7fcc696f795ab337"
server_key_version: "encryptionkey@0"
I20260810 18:10:31.467402 11263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 18:10:31.470458 11279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:31.472142 11263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 18:10:31.472303 11263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "9a8f5609d62d44b9ae5b248754f25527"
format_stamp: "Formatted at 2026-08-10 18:10:31 on dist-test-slave-27sr"
server_key: "bf380a219a168f364fc15cc94152de0c"
server_key_iv: "fd572bf3c17ee53a7fcc696f795ab337"
server_key_version: "encryptionkey@0"
I20260810 18:10:31.472443 11263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:31.495814 11263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:31.496644 11263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:31.496879 11263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:31.497548 11263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:31.498596 11263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:31.498674 11263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:31.498759 11263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:31.498803 11263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:31.508872 11263 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:45333
I20260810 18:10:31.508893 11415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:45333 every 8 connection(s)
I20260810 18:10:31.509883 11263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:10:31.514842 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 11263
I20260810 18:10:31.515017 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:10:31.515336 22233 external_mini_cluster.cc:1428] Setting key 9512200bb03ca51c65eb76e36b78f426
I20260810 18:10:31.523427 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.511880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:52843 (local address 127.21.182.126:40581)
0810 18:10:31.512111 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:31.512115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:31.513328 (+ 1213us) server_negotiation.cc:407] Connection header received
0810 18:10:31.514862 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:31.514865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:31.514963 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:31.515052 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:31.516400 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.517233 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.519318 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.519545 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.519643 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.522700 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:31.522705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:31.522708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:31.522725 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:31.522756 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:31.522765 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:31.522854 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:31.523135 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 18:10:31.523235 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260810 18:10:31.524958 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.512363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:40581 (local address 127.21.182.67:52843)
0810 18:10:31.513138 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:31.513182 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:31.514582 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:31.515290 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:31.515302 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:31.515406 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:31.516217 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.516241 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.517424 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.517427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.519159 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.519171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.519723 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.519736 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.520032 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.522358 (+ 2326us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:10:31.522360 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:31.522412 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:10:31.522525 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:31.522925 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:31.522938 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:31.522947 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:10:31.523616 (+ 669us) client_negotiation.cc:240] Negotiation successful
0810 18:10:31.524066 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":111,"threads_started":1}
I20260810 18:10:31.526216 11416 heartbeater.cc:347] Connected to a master server at 127.21.182.126:40581
I20260810 18:10:31.526522 11416 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:31.527346 11416 heartbeater.cc:511] Master 127.21.182.126:40581 requested a full tablet report, sending...
I20260810 18:10:31.530346 10887 ts_manager.cc:194] Registered new tserver with Master: 9a8f5609d62d44b9ae5b248754f25527 (127.21.182.67:45333)
I20260810 18:10:31.531023 10887 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.182.67:52843
I20260810 18:10:31.542858 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:10:31.550905 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.545492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.21.182.126:40581)
0810 18:10:31.545630 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:31.545635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:31.545815 (+ 180us) server_negotiation.cc:407] Connection header received
0810 18:10:31.545918 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:31.545920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:31.545967 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:31.546086 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:31.546455 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.547071 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.548090 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.548419 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.548487 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.549953 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:31.549960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:31.549966 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:31.549989 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:31.550020 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:31.550035 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:31.550209 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:31.550634 (+ 425us) server_negotiation.cc:299] Negotiation successful
0810 18:10:31.550687 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 18:10:31.558180 10887 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:10:31.564126 10887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:10:31.584308 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.577661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60962 (local address 127.21.182.67:45333)
0810 18:10:31.578062 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:31.578076 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:31.578103 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:10:31.578187 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:31.578195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:31.578431 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:31.578632 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:31.578986 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.580102 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.582056 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.583137 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.583250 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.583478 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:31.583950 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 18:10:31.584084 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":151,"threads_started":1}
I20260810 18:10:31.584723 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.578710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:39709 (local address 127.0.0.1:43202)
0810 18:10:31.578985 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:31.579000 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:31.579084 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:31.580089 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:31.580092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:31.580114 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:31.580203 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.580210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.581585 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.581589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.583053 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.583066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.584229 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.584233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.584317 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.584347 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 18:10:31.584403 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 18:10:31.584561 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":81,"thread_start_us":127,"threads_started":1}
I20260810 18:10:31.584791 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.576807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:35825 (local address 127.0.0.1:40850)
0810 18:10:31.578693 (+ 1886us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:31.578709 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:31.578813 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:31.579665 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:31.579669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:31.579691 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:31.579803 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.579813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.581544 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.581548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.583053 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.583066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.584313 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.584319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.584445 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.584485 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 18:10:31.584564 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 18:10:31.584640 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1794,"mutex_wait_us":67,"thread_start_us":113,"threads_started":1}
I20260810 18:10:31.585028 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.576750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:45333 (local address 127.0.0.1:60962)
0810 18:10:31.577308 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:31.577377 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:31.577546 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:31.578624 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:31.578631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:31.578671 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:31.578837 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.578846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.580232 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.580236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.581860 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.581871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.583548 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.583551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.583668 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.583709 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 18:10:31.584811 (+ 1102us) client_negotiation.cc:240] Negotiation successful
0810 18:10:31.584898 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":93,"threads_started":1}
I20260810 18:10:31.585532 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.577820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.21.182.65:35825)
0810 18:10:31.579100 (+ 1280us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:31.579107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:31.579136 (+ 29us) server_negotiation.cc:407] Connection header received
0810 18:10:31.579236 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:31.579242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:31.579477 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:31.579687 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:31.579952 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.581409 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.583240 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.583868 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.584011 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.584233 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:31.585207 (+ 974us) server_negotiation.cc:299] Negotiation successful
0810 18:10:31.585338 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1148,"thread_start_us":114,"threads_started":1}
I20260810 18:10:31.585597 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.579220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43202 (local address 127.21.182.66:39709)
0810 18:10:31.579611 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:31.579624 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:31.579648 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:10:31.579719 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:31.579724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:31.579944 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:31.580090 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:31.580339 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.581442 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.583192 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.584094 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.585156 (+ 1062us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.585357 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:31.585402 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 18:10:31.585474 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":164,"threads_started":1}
I20260810 18:10:31.588639 11183 tablet_service.cc:1467] Processing CreateTablet for tablet 7bfd66999099419b935d265377edc7ad (DEFAULT_TABLE table=test-table [id=8c37ca0ca1d34fd98b4d0fc97e7a8edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:31.588639 11336 tablet_service.cc:1467] Processing CreateTablet for tablet 7bfd66999099419b935d265377edc7ad (DEFAULT_TABLE table=test-table [id=8c37ca0ca1d34fd98b4d0fc97e7a8edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:31.588639 11027 tablet_service.cc:1467] Processing CreateTablet for tablet 7bfd66999099419b935d265377edc7ad (DEFAULT_TABLE table=test-table [id=8c37ca0ca1d34fd98b4d0fc97e7a8edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:31.589798 11183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bfd66999099419b935d265377edc7ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:31.589800 11027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bfd66999099419b935d265377edc7ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:31.589800 11336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bfd66999099419b935d265377edc7ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:31.597887 11436 tablet_bootstrap.cc:492] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917: Bootstrap starting.
I20260810 18:10:31.599282 11438 tablet_bootstrap.cc:492] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527: Bootstrap starting.
I20260810 18:10:31.600282 11437 tablet_bootstrap.cc:492] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205: Bootstrap starting.
I20260810 18:10:31.600769 11436 tablet_bootstrap.cc:654] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:31.601588 11438 tablet_bootstrap.cc:654] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:31.601799 11436 log.cc:824] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:31.602505 11438 log.cc:824] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:31.602562 11437 tablet_bootstrap.cc:654] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:31.603576 11437 log.cc:824] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:31.604629 11436 tablet_bootstrap.cc:492] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917: No bootstrap required, opened a new log
I20260810 18:10:31.604715 11438 tablet_bootstrap.cc:492] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527: No bootstrap required, opened a new log
I20260810 18:10:31.604892 11436 ts_tablet_manager.cc:1397] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:10:31.604923 11438 ts_tablet_manager.cc:1397] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:10:31.605715 11437 tablet_bootstrap.cc:492] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205: No bootstrap required, opened a new log
I20260810 18:10:31.605913 11437 ts_tablet_manager.cc:1397] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:10:31.609297 11436 raft_consensus.cc:348] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8f5609d62d44b9ae5b248754f25527" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45333 } } peers { permanent_uuid: "eea1a9e285f74c8681e6230d6be4c205" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 35825 } } peers { permanent_uuid: "e30c35b03d7946eaa93184e82e6db917" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39709 } }
I20260810 18:10:31.609609 11436 raft_consensus.cc:374] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:31.609448 11438 raft_consensus.cc:348] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8f5609d62d44b9ae5b248754f25527" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45333 } } peers { permanent_uuid: "eea1a9e285f74c8681e6230d6be4c205" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 35825 } } peers { permanent_uuid: "e30c35b03d7946eaa93184e82e6db917" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39709 } }
I20260810 18:10:31.609678 11436 raft_consensus.cc:729] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30c35b03d7946eaa93184e82e6db917, State: Initialized, Role: FOLLOWER
I20260810 18:10:31.609731 11438 raft_consensus.cc:374] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:31.609808 11438 raft_consensus.cc:729] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8f5609d62d44b9ae5b248754f25527, State: Initialized, Role: FOLLOWER
I20260810 18:10:31.610235 11436 consensus_queue.cc:260] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8f5609d62d44b9ae5b248754f25527" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45333 } } peers { permanent_uuid: "eea1a9e285f74c8681e6230d6be4c205" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 35825 } } peers { permanent_uuid: "e30c35b03d7946eaa93184e82e6db917" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39709 } }
I20260810 18:10:31.610275 11438 consensus_queue.cc:260] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8f5609d62d44b9ae5b248754f25527" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45333 } } peers { permanent_uuid: "eea1a9e285f74c8681e6230d6be4c205" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 35825 } } peers { permanent_uuid: "e30c35b03d7946eaa93184e82e6db917" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39709 } }
I20260810 18:10:31.610236 11437 raft_consensus.cc:348] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8f5609d62d44b9ae5b248754f25527" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45333 } } peers { permanent_uuid: "eea1a9e285f74c8681e6230d6be4c205" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 35825 } } peers { permanent_uuid: "e30c35b03d7946eaa93184e82e6db917" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39709 } }
I20260810 18:10:31.610512 11437 raft_consensus.cc:374] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:31.610584 11437 raft_consensus.cc:729] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea1a9e285f74c8681e6230d6be4c205, State: Initialized, Role: FOLLOWER
I20260810 18:10:31.610996 11437 consensus_queue.cc:260] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8f5609d62d44b9ae5b248754f25527" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45333 } } peers { permanent_uuid: "eea1a9e285f74c8681e6230d6be4c205" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 35825 } } peers { permanent_uuid: "e30c35b03d7946eaa93184e82e6db917" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39709 } }
I20260810 18:10:31.611413 11438 ts_tablet_manager.cc:1428] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:10:31.611440 11436 ts_tablet_manager.cc:1428] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:10:31.611819 11260 heartbeater.cc:503] Master 127.21.182.126:40581 was elected leader, sending a full tablet report...
I20260810 18:10:31.611819 11107 heartbeater.cc:503] Master 127.21.182.126:40581 was elected leader, sending a full tablet report...
I20260810 18:10:31.613168 11416 heartbeater.cc:503] Master 127.21.182.126:40581 was elected leader, sending a full tablet report...
I20260810 18:10:31.614490 11437 ts_tablet_manager.cc:1428] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
W20260810 18:10:31.760919 11417 tablet.cc:2369] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:10:31.760947 11442 raft_consensus.cc:482] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:10:31.761183 11442 raft_consensus.cc:504] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8f5609d62d44b9ae5b248754f25527" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45333 } } peers { permanent_uuid: "eea1a9e285f74c8681e6230d6be4c205" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 35825 } } peers { permanent_uuid: "e30c35b03d7946eaa93184e82e6db917" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39709 } }
I20260810 18:10:31.762552 11442 leader_election.cc:288] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a8f5609d62d44b9ae5b248754f25527 (127.21.182.67:45333), eea1a9e285f74c8681e6230d6be4c205 (127.21.182.65:35825)
I20260810 18:10:31.768635 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.763025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:33265 (local address 127.21.182.65:35825)
0810 18:10:31.763151 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:31.763156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:31.763401 (+ 245us) server_negotiation.cc:407] Connection header received
0810 18:10:31.763496 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:31.763501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:31.763556 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:31.763631 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:31.763924 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.764736 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.766892 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.767608 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.767733 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.767793 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:31.768371 (+ 578us) server_negotiation.cc:299] Negotiation successful
0810 18:10:31.768471 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 18:10:31.768682 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.762790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:45333 (local address 127.21.182.66:49463)
0810 18:10:31.762979 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:31.763001 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:31.763116 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:31.763562 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:31.763564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:31.763580 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:31.763749 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.763761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.764857 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.764861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.766633 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.766649 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.767748 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.767753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.768082 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.768202 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 18:10:31.768301 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 18:10:31.768464 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":367}
I20260810 18:10:31.768635 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.763167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:49463 (local address 127.21.182.67:45333)
0810 18:10:31.763275 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:31.763279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:31.763300 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:10:31.763354 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:31.763357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:31.763437 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:31.763528 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:31.763934 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.764707 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.766892 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.767589 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.767826 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.767867 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:31.768379 (+ 512us) server_negotiation.cc:299] Negotiation successful
0810 18:10:31.768481 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 18:10:31.768692 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.762916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:35825 (local address 127.21.182.66:33265)
0810 18:10:31.763305 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:31.763318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:31.763413 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:31.763657 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:31.763660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:31.763674 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:31.763786 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.763796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.764857 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.764861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.766633 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:31.766649 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.767748 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.767753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:31.768082 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.768204 (+ 122us) client_negotiation.cc:769] Sending connection context
0810 18:10:31.768301 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 18:10:31.768464 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":380,"thread_start_us":84,"threads_started":1}
I20260810 18:10:31.769595 11361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bfd66999099419b935d265377edc7ad" candidate_uuid: "e30c35b03d7946eaa93184e82e6db917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a8f5609d62d44b9ae5b248754f25527" is_pre_election: true
I20260810 18:10:31.769814 11052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bfd66999099419b935d265377edc7ad" candidate_uuid: "e30c35b03d7946eaa93184e82e6db917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea1a9e285f74c8681e6230d6be4c205" is_pre_election: true
I20260810 18:10:31.769976 11361 raft_consensus.cc:2454] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30c35b03d7946eaa93184e82e6db917 in term 0.
I20260810 18:10:31.770179 11052 raft_consensus.cc:2454] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30c35b03d7946eaa93184e82e6db917 in term 0.
I20260810 18:10:31.770619 11131 leader_election.cc:302] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a8f5609d62d44b9ae5b248754f25527, e30c35b03d7946eaa93184e82e6db917; no voters:
I20260810 18:10:31.770988 11442 raft_consensus.cc:2781] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:10:31.771103 11442 raft_consensus.cc:482] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:10:31.771178 11442 raft_consensus.cc:3037] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:31.772429 11442 raft_consensus.cc:504] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8f5609d62d44b9ae5b248754f25527" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45333 } } peers { permanent_uuid: "eea1a9e285f74c8681e6230d6be4c205" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 35825 } } peers { permanent_uuid: "e30c35b03d7946eaa93184e82e6db917" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39709 } }
I20260810 18:10:31.772852 11442 leader_election.cc:288] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [CANDIDATE]: Term 1 election: Requested vote from peers 9a8f5609d62d44b9ae5b248754f25527 (127.21.182.67:45333), eea1a9e285f74c8681e6230d6be4c205 (127.21.182.65:35825)
I20260810 18:10:31.773288 11052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bfd66999099419b935d265377edc7ad" candidate_uuid: "e30c35b03d7946eaa93184e82e6db917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea1a9e285f74c8681e6230d6be4c205"
I20260810 18:10:31.773322 11361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bfd66999099419b935d265377edc7ad" candidate_uuid: "e30c35b03d7946eaa93184e82e6db917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a8f5609d62d44b9ae5b248754f25527"
I20260810 18:10:31.773454 11052 raft_consensus.cc:3037] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:31.773486 11361 raft_consensus.cc:3037] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:31.774677 11361 raft_consensus.cc:2454] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30c35b03d7946eaa93184e82e6db917 in term 1.
I20260810 18:10:31.774677 11052 raft_consensus.cc:2454] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30c35b03d7946eaa93184e82e6db917 in term 1.
I20260810 18:10:31.775053 11131 leader_election.cc:302] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a8f5609d62d44b9ae5b248754f25527, e30c35b03d7946eaa93184e82e6db917; no voters:
I20260810 18:10:31.775278 11442 raft_consensus.cc:2781] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:31.775592 11442 raft_consensus.cc:686] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [term 1 LEADER]: Becoming Leader. State: Replica: e30c35b03d7946eaa93184e82e6db917, State: Running, Role: LEADER
I20260810 18:10:31.776067 11442 consensus_queue.cc:237] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8f5609d62d44b9ae5b248754f25527" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45333 } } peers { permanent_uuid: "eea1a9e285f74c8681e6230d6be4c205" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 35825 } } peers { permanent_uuid: "e30c35b03d7946eaa93184e82e6db917" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39709 } }
I20260810 18:10:31.780294 10883 catalog_manager.cc:5495] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 reported cstate change: term changed from 0 to 1, leader changed from <none> to e30c35b03d7946eaa93184e82e6db917 (127.21.182.66). New cstate: current_term: 1 leader_uuid: "e30c35b03d7946eaa93184e82e6db917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8f5609d62d44b9ae5b248754f25527" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea1a9e285f74c8681e6230d6be4c205" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 35825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30c35b03d7946eaa93184e82e6db917" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 39709 } health_report { overall_health: HEALTHY } } }
W20260810 18:10:31.795423 11261 tablet.cc:2369] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:10:31.833148 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.828915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.21.182.66:39709)
0810 18:10:31.829093 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:31.829098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:31.829111 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:10:31.829202 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:31.829205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:31.829268 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:31.829405 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:10:31.829710 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.830226 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.831419 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:31.831638 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:31.831724 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:31.832189 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:10:31.832641 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:10:31.832768 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:31.832956 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 18:10:31.832997 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 18:10:31.837445 11108 tablet.cc:2369] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:10:31.839835 11361 raft_consensus.cc:1261] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527 [term 1 FOLLOWER]: Refusing update from remote peer e30c35b03d7946eaa93184e82e6db917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:10:31.839967 11052 raft_consensus.cc:1261] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205 [term 1 FOLLOWER]: Refusing update from remote peer e30c35b03d7946eaa93184e82e6db917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:10:31.840723 11442 consensus_queue.cc:1035] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a8f5609d62d44b9ae5b248754f25527" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:10:31.840960 11447 consensus_queue.cc:1035] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea1a9e285f74c8681e6230d6be4c205" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:10:31.850860 11451 mvcc.cc:205] Tried to move back new op lower bound from 7317034728807100416 to 7317034728559329280. Current Snapshot: MvccSnapshot[applied={T|T < 7317034728807100416}]
I20260810 18:10:31.852258 11452 mvcc.cc:205] Tried to move back new op lower bound from 7317034728807100416 to 7317034728559329280. Current Snapshot: MvccSnapshot[applied={T|T < 7317034728807100416}]
I20260810 18:10:31.858050 10882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:10:31.858208 10882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:10:31.860882 10882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b554b96694e4000ab1cf229f1a48e91: Sending DeleteTablet for 3 replicas of tablet 7bfd66999099419b935d265377edc7ad
I20260810 18:10:31.861780 11027 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bfd66999099419b935d265377edc7ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:31 UTC) from {username='slave'} at 127.0.0.1:40850
I20260810 18:10:31.861820 11336 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bfd66999099419b935d265377edc7ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:31 UTC) from {username='slave'} at 127.0.0.1:60962
I20260810 18:10:31.861917 11183 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bfd66999099419b935d265377edc7ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:31 UTC) from {username='slave'} at 127.0.0.1:43202
I20260810 18:10:31.862352 11463 tablet_replica.cc:330] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527: stopping tablet replica
I20260810 18:10:31.862583 11461 tablet_replica.cc:330] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205: stopping tablet replica
I20260810 18:10:31.862939 11461 raft_consensus.cc:2229] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:10:31.863075 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 10955
I20260810 18:10:31.863308 11461 raft_consensus.cc:2258] T 7bfd66999099419b935d265377edc7ad P eea1a9e285f74c8681e6230d6be4c205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:31.868381 11464 tablet_replica.cc:330] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917: stopping tablet replica
I20260810 18:10:31.868355 11463 raft_consensus.cc:2229] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:10:31.868752 11464 raft_consensus.cc:2229] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:10:31.868757 11463 raft_consensus.cc:2258] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:31.869189 11464 raft_consensus.cc:2258] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:31.870683 11463 ts_tablet_manager.cc:1905] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:10:31.870745 11464 ts_tablet_manager.cc:1905] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:10:31.873973 11464 ts_tablet_manager.cc:1918] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:10:31.874087 11464 log.cc:1196] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/7bfd66999099419b935d265377edc7ad
I20260810 18:10:31.874296 11463 ts_tablet_manager.cc:1918] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:10:31.874419 11463 log.cc:1196] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/7bfd66999099419b935d265377edc7ad
I20260810 18:10:31.874452 11464 ts_tablet_manager.cc:1939] T 7bfd66999099419b935d265377edc7ad P e30c35b03d7946eaa93184e82e6db917: Deleting consensus metadata
I20260810 18:10:31.874773 11463 ts_tablet_manager.cc:1939] T 7bfd66999099419b935d265377edc7ad P 9a8f5609d62d44b9ae5b248754f25527: Deleting consensus metadata
I20260810 18:10:31.875931 10866 catalog_manager.cc:4841] TS e30c35b03d7946eaa93184e82e6db917 (127.21.182.66:39709): tablet 7bfd66999099419b935d265377edc7ad (table test-table [id=8c37ca0ca1d34fd98b4d0fc97e7a8edd]) successfully deleted
I20260810 18:10:31.876183 10866 catalog_manager.cc:4841] TS 9a8f5609d62d44b9ae5b248754f25527 (127.21.182.67:45333): tablet 7bfd66999099419b935d265377edc7ad (table test-table [id=8c37ca0ca1d34fd98b4d0fc97e7a8edd]) successfully deleted
W20260810 18:10:31.876843 10867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:35825 (error 108)
I20260810 18:10:31.877056 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 11110
I20260810 18:10:31.881701 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:31.881317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:35825 (local address 127.0.0.1:40856)
0810 18:10:31.881477 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:31.881571 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:35825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260810 18:10:31.881948 10867 catalog_manager.cc:4584] TS eea1a9e285f74c8681e6230d6be4c205 (127.21.182.65:35825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bfd66999099419b935d265377edc7ad: Network error: Client connection negotiation failed: client connection to 127.21.182.65:35825: connect: Connection refused (error 111)
I20260810 18:10:31.887657 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 11263
I20260810 18:10:31.897137 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 10850
2026-08-10T18:10:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1351 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:31 dist-test-slave-27sr krb5kdc[11468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:31 dist-test-slave-27sr krb5kdc[11468](info): set up 2 sockets
Aug 10 18:10:31 dist-test-slave-27sr krb5kdc[11468](info): commencing operation
krb5kdc: starting...
W20260810 18:10:33.079002 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:33 dist-test-slave-27sr krb5kdc[11468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:10:33.220148 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:35145
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:35145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:33.343678 11488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:33.344110 11488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:33.344204 11488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:33.349066 11488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:33.349140 11488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:33.349165 11488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:33.349185 11488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:33.349203 11488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:33.355625 11488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:35145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:35145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:33.357029 11488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:33.358276 11488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:33.368211 11493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:33.368211 11496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:33.368281 11494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:33.368584 11488 server_base.cc:1034] running on GCE node
I20260810 18:10:33.369299 11488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:33.370575 11488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:33.371780 11488 hybrid_clock.cc:648] HybridClock initialized: now 1786385433371755 us; error 37 us; skew 500 ppm
Aug 10 18:10:33 dist-test-slave-27sr krb5kdc[11468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385433, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:33.374802 11488 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:10:33.375744 11488 webserver.cc:460] Webserver started at http://127.21.182.126:43185/ using document root <none> and password file <none>
I20260810 18:10:33.376420 11488 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:33.376521 11488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:33.376773 11488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:33.379017 11488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "82d872f71976405f955f6cdeb633169a"
format_stamp: "Formatted at 2026-08-10 18:10:33 on dist-test-slave-27sr"
server_key: "8c1d3c7cc654fef303cd7a9be5f966ea"
server_key_iv: "6319eebabd77b45c55d79859fa54c09a"
server_key_version: "encryptionkey@0"
I20260810 18:10:33.379623 11488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "82d872f71976405f955f6cdeb633169a"
format_stamp: "Formatted at 2026-08-10 18:10:33 on dist-test-slave-27sr"
server_key: "8c1d3c7cc654fef303cd7a9be5f966ea"
server_key_iv: "6319eebabd77b45c55d79859fa54c09a"
server_key_version: "encryptionkey@0"
I20260810 18:10:33.383731 11488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:10:33.386446 11505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:33.387724 11488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:10:33.387950 11488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "82d872f71976405f955f6cdeb633169a"
format_stamp: "Formatted at 2026-08-10 18:10:33 on dist-test-slave-27sr"
server_key: "8c1d3c7cc654fef303cd7a9be5f966ea"
server_key_iv: "6319eebabd77b45c55d79859fa54c09a"
server_key_version: "encryptionkey@0"
I20260810 18:10:33.388099 11488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:33.397874 11488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:33.401315 11488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:33.401536 11488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:33.408914 11488 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:35145
I20260810 18:10:33.408962 11569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:35145 every 8 connection(s)
I20260810 18:10:33.409956 11488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:33.412814 11570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:33.418334 11570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a: Bootstrap starting.
I20260810 18:10:33.418694 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 11488
I20260810 18:10:33.418844 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:33.419147 22233 external_mini_cluster.cc:1428] Setting key a6371656ec7ed4d929e750b1cfd34cc0
I20260810 18:10:33.421212 11570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:33.421998 11570 log.cc:824] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:33.424423 11570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a: No bootstrap required, opened a new log
Aug 10 18:10:33 dist-test-slave-27sr krb5kdc[11468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:10:33.429394 11570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d872f71976405f955f6cdeb633169a" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35145 } }
I20260810 18:10:33.429746 11570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:33.429847 11570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d872f71976405f955f6cdeb633169a, State: Initialized, Role: FOLLOWER
I20260810 18:10:33.430438 11570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d872f71976405f955f6cdeb633169a" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35145 } }
I20260810 18:10:33.430634 11570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:33.430730 11570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:33.430855 11570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:33.432255 11570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d872f71976405f955f6cdeb633169a" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35145 } }
I20260810 18:10:33.432698 11570 leader_election.cc:302] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82d872f71976405f955f6cdeb633169a; no voters:
I20260810 18:10:33.433117 11570 leader_election.cc:288] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:33.433257 11575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:33.433521 11575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [term 1 LEADER]: Becoming Leader. State: Replica: 82d872f71976405f955f6cdeb633169a, State: Running, Role: LEADER
I20260810 18:10:33.433918 11575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d872f71976405f955f6cdeb633169a" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35145 } }
I20260810 18:10:33.434698 11570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:33.435817 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:33.420567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.21.182.126:35145)
0810 18:10:33.421020 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:33.421041 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:33.421078 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:10:33.421828 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:33.421849 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:33.422288 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:33.422683 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:33.423654 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:33.424772 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:33.425953 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:33.426444 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:33.426569 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:33.429679 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:33.429686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:33.429707 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:33.429746 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:33.432025 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:33.432625 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:33.432632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:33.432638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:33.432803 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:33.433175 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:33.433180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:33.433182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:33.433813 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:33.434056 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:33.434361 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 18:10:33.434696 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":152,"threads_started":1}
I20260810 18:10:33.436069 11578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82d872f71976405f955f6cdeb633169a. Latest consensus state: current_term: 1 leader_uuid: "82d872f71976405f955f6cdeb633169a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d872f71976405f955f6cdeb633169a" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35145 } } }
I20260810 18:10:33.436204 11578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:33.435855 11576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82d872f71976405f955f6cdeb633169a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d872f71976405f955f6cdeb633169a" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 35145 } } }
I20260810 18:10:33.436398 11576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a [sys.catalog]: This master's current role is: LEADER
W20260810 18:10:33.441174 11592 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:10:33.441293 11592 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:10:33.441387 11586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:33.442174 11586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:33.448900 11586 catalog_manager.cc:1347] Generated new cluster ID: 9b0ce6035f45418db43c3706533a44ce
I20260810 18:10:33.448997 11586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:33.465659 11586 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:33.466701 11586 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:33.474267 11586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a: Generated new TSK 0
I20260810 18:10:33.475162 11586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.182.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
Entry for principal kudu/127.21.182.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab.
I20260810 18:10:33.511590 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35145
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:33.631791 11600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:33.632193 11600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:33.632239 11600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:33.636693 11600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:33.636766 11600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:33.636828 11600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:10:33.642586 11600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.65.keytab
--principal=kudu/127.21.182.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:33.643759 11600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:33.645076 11600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:33.655825 11605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:33.655982 11606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:33.655982 11611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:33.656204 11600 server_base.cc:1034] running on GCE node
I20260810 18:10:33.656630 11600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:33.657263 11600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:33.658478 11600 hybrid_clock.cc:648] HybridClock initialized: now 1786385433658459 us; error 35 us; skew 500 ppm
Aug 10 18:10:33 dist-test-slave-27sr krb5kdc[11468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385433, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:33.661599 11600 init.cc:377] Logged in from keytab as kudu/127.21.182.65@KRBTEST.COM (short username kudu)
I20260810 18:10:33.662581 11600 webserver.cc:460] Webserver started at http://127.21.182.65:38783/ using document root <none> and password file <none>
I20260810 18:10:33.663183 11600 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:33.663288 11600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:33.663533 11600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:33.665724 11600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "194af3d5682b40f28953bcebd14d45d9"
format_stamp: "Formatted at 2026-08-10 18:10:33 on dist-test-slave-27sr"
server_key: "d0b4d5d59b34b29032e893ac47f70084"
server_key_iv: "5357da009beee17031d8c0f87403a113"
server_key_version: "encryptionkey@0"
I20260810 18:10:33.666322 11600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "194af3d5682b40f28953bcebd14d45d9"
format_stamp: "Formatted at 2026-08-10 18:10:33 on dist-test-slave-27sr"
server_key: "d0b4d5d59b34b29032e893ac47f70084"
server_key_iv: "5357da009beee17031d8c0f87403a113"
server_key_version: "encryptionkey@0"
I20260810 18:10:33.670406 11600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:10:33.673354 11619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:33.674885 11600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:33.675062 11600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "194af3d5682b40f28953bcebd14d45d9"
format_stamp: "Formatted at 2026-08-10 18:10:33 on dist-test-slave-27sr"
server_key: "d0b4d5d59b34b29032e893ac47f70084"
server_key_iv: "5357da009beee17031d8c0f87403a113"
server_key_version: "encryptionkey@0"
I20260810 18:10:33.675200 11600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:33.694625 11600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:33.698072 11600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:33.698328 11600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:33.699052 11600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:33.700115 11600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:33.700205 11600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:33.700292 11600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:33.700352 11600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:33.710083 11600 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:39893
I20260810 18:10:33.710110 11755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:39893 every 8 connection(s)
I20260810 18:10:33.711128 11600 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:10:33.719626 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 11600
I20260810 18:10:33.719832 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:10:33.720191 22233 external_mini_cluster.cc:1428] Setting key fa9effffb11e98ba18c2b9866ddd2aae
Aug 10 18:10:33 dist-test-slave-27sr krb5kdc[11468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385433, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.65@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:10:33.727389 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:33.714961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:56389 (local address 127.21.182.126:35145)
0810 18:10:33.715153 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:33.715159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:33.715183 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:10:33.716215 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:33.716219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:33.716277 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:33.716359 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:33.717587 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:33.718131 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:33.719359 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:33.719604 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:33.719728 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:33.722835 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:33.722841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:33.722847 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:33.722891 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:33.724749 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:33.725450 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:33.725452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:33.725454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:33.725533 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:33.725975 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:33.725982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:33.725986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:33.726248 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:33.726357 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:33.726960 (+ 603us) server_negotiation.cc:299] Negotiation successful
0810 18:10:33.727123 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 18:10:33.728274 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:33.714027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35145 (local address 127.21.182.65:56389)
0810 18:10:33.714547 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:33.714595 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:33.715986 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:33.716545 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:33.716552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:33.716935 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:33.717433 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:33.717447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:33.718270 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:33.718274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:33.719205 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:33.719215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:33.719769 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:33.719773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:33.720016 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:33.720686 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:10:33.720704 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:33.722643 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:33.724880 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:33.724890 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:33.724913 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:33.725338 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:33.725636 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:33.725640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:33.725642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:33.725856 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:33.726376 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:33.726382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:33.726664 (+ 282us) client_negotiation.cc:769] Sending connection context
0810 18:10:33.726866 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 18:10:33.727171 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":140,"threads_started":1}
I20260810 18:10:33.729862 11756 heartbeater.cc:347] Connected to a master server at 127.21.182.126:35145
I20260810 18:10:33.730151 11756 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:33.730700 11756 heartbeater.cc:511] Master 127.21.182.126:35145 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.182.66@KRBTEST.COM; defaulting to no policy
I20260810 18:10:33.732601 11525 ts_manager.cc:194] Registered new tserver with Master: 194af3d5682b40f28953bcebd14d45d9 (127.21.182.65:39893)
I20260810 18:10:33.733877 11525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.65@KRBTEST.COM'} at 127.21.182.65:56389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
Entry for principal kudu/127.21.182.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab.
I20260810 18:10:33.755767 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35145
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:33.877405 11763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:33.877723 11763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:33.877768 11763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:33.882100 11763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:33.882184 11763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:33.882249 11763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:10:33.888073 11763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.66.keytab
--principal=kudu/127.21.182.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:33.889259 11763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:33.890365 11763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:33.901535 11771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:33.901530 11770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:33.901640 11773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:33.901881 11763 server_base.cc:1034] running on GCE node
I20260810 18:10:33.902472 11763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:33.903294 11763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:33.904465 11763 hybrid_clock.cc:648] HybridClock initialized: now 1786385433904438 us; error 39 us; skew 500 ppm
Aug 10 18:10:33 dist-test-slave-27sr krb5kdc[11468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385433, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:33.907866 11763 init.cc:377] Logged in from keytab as kudu/127.21.182.66@KRBTEST.COM (short username kudu)
I20260810 18:10:33.909019 11763 webserver.cc:460] Webserver started at http://127.21.182.66:38517/ using document root <none> and password file <none>
I20260810 18:10:33.909726 11763 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:33.909833 11763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:33.910102 11763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:33.912505 11763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "8b0f4d0321d24d29844e14431d364e7f"
format_stamp: "Formatted at 2026-08-10 18:10:33 on dist-test-slave-27sr"
server_key: "0c6d99b86c909a8185fdaa43542dd08a"
server_key_iv: "bceed61279d20b3975cdeed7de7c7914"
server_key_version: "encryptionkey@0"
I20260810 18:10:33.913153 11763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "8b0f4d0321d24d29844e14431d364e7f"
format_stamp: "Formatted at 2026-08-10 18:10:33 on dist-test-slave-27sr"
server_key: "0c6d99b86c909a8185fdaa43542dd08a"
server_key_iv: "bceed61279d20b3975cdeed7de7c7914"
server_key_version: "encryptionkey@0"
I20260810 18:10:33.917555 11763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:10:33.920503 11781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:33.922025 11763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:33.922185 11763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "8b0f4d0321d24d29844e14431d364e7f"
format_stamp: "Formatted at 2026-08-10 18:10:33 on dist-test-slave-27sr"
server_key: "0c6d99b86c909a8185fdaa43542dd08a"
server_key_iv: "bceed61279d20b3975cdeed7de7c7914"
server_key_version: "encryptionkey@0"
I20260810 18:10:33.922317 11763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:33.945430 11763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:33.949085 11763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:33.949326 11763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:33.950012 11763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:33.950976 11763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:33.951064 11763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:33.951159 11763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:33.951210 11763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:33.961099 11763 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:45083
I20260810 18:10:33.961130 11918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:45083 every 8 connection(s)
I20260810 18:10:33.962181 11763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:10:33.964571 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 11763
I20260810 18:10:33.964829 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:10:33.965181 22233 external_mini_cluster.cc:1428] Setting key 2647b39246bab0abafd780697e07faa0
Aug 10 18:10:33 dist-test-slave-27sr krb5kdc[11468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385433, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.66@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.67@KRBTEST.COM; defaulting to no policy
I20260810 18:10:33.978739 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:33.964520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:50843 (local address 127.21.182.126:35145)
0810 18:10:33.964709 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:33.964714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:33.965801 (+ 1087us) server_negotiation.cc:407] Connection header received
0810 18:10:33.967432 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:33.967436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:33.967497 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:33.967579 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:33.969051 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:33.969571 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:33.971027 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:33.971260 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:33.971377 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:33.974761 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:33.974764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:33.974766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:33.974795 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:33.976416 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:33.977105 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:33.977108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:33.977110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:33.977194 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:33.977529 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:33.977531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:33.977532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:33.977723 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:33.977805 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:33.978469 (+ 664us) server_negotiation.cc:299] Negotiation successful
0810 18:10:33.978590 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 18:10:33.980001 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:33.965075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35145 (local address 127.21.182.66:50843)
0810 18:10:33.965625 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:33.965688 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:33.967148 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:33.967780 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:33.967787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:33.968253 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:33.968852 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:33.968876 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:33.969715 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:33.969722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:33.970889 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:33.970900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:33.971443 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:33.971448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:33.971627 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:33.972453 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:10:33.972472 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:33.974552 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:33.976554 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:33.976562 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:33.976582 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:33.976991 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:33.977288 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:33.977290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:33.977292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:33.977443 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:33.977852 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:33.977858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:33.978207 (+ 349us) client_negotiation.cc:769] Sending connection context
0810 18:10:33.978398 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 18:10:33.978745 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":166,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.67@KRBTEST.COM" created.
I20260810 18:10:33.981571 11919 heartbeater.cc:347] Connected to a master server at 127.21.182.126:35145
I20260810 18:10:33.981942 11919 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:33.982522 11919 heartbeater.cc:511] Master 127.21.182.126:35145 requested a full tablet report, sending...
I20260810 18:10:33.984010 11525 ts_manager.cc:194] Registered new tserver with Master: 8b0f4d0321d24d29844e14431d364e7f (127.21.182.66:45083)
I20260810 18:10:33.984653 11525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.66@KRBTEST.COM'} at 127.21.182.66:50843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
Entry for principal kudu/127.21.182.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab.
I20260810 18:10:33.999764 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35145
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:34.124094 11924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:34.124411 11924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:34.124466 11924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:34.128973 11924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:34.129051 11924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:34.129160 11924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:10:34.135115 11924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.67.keytab
--principal=kudu/127.21.182.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:34.136493 11924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:34.137751 11924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:34.148388 11933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:34.148382 11931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:34.148388 11930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:34.148433 11924 server_base.cc:1034] running on GCE node
I20260810 18:10:34.149142 11924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:34.149897 11924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:34.151091 11924 hybrid_clock.cc:648] HybridClock initialized: now 1786385434151058 us; error 45 us; skew 500 ppm
Aug 10 18:10:34 dist-test-slave-27sr krb5kdc[11468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385434, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:34.154197 11924 init.cc:377] Logged in from keytab as kudu/127.21.182.67@KRBTEST.COM (short username kudu)
I20260810 18:10:34.155253 11924 webserver.cc:460] Webserver started at http://127.21.182.67:39205/ using document root <none> and password file <none>
I20260810 18:10:34.155885 11924 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:34.156041 11924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:34.156294 11924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:34.158569 11924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "62b83a4acae34e81bde81abf8ac8466c"
format_stamp: "Formatted at 2026-08-10 18:10:34 on dist-test-slave-27sr"
server_key: "d6c139e70a7765225ff3812c641ea39b"
server_key_iv: "f5998fd8c4c5299924f6debaf764c918"
server_key_version: "encryptionkey@0"
I20260810 18:10:34.159204 11924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "62b83a4acae34e81bde81abf8ac8466c"
format_stamp: "Formatted at 2026-08-10 18:10:34 on dist-test-slave-27sr"
server_key: "d6c139e70a7765225ff3812c641ea39b"
server_key_iv: "f5998fd8c4c5299924f6debaf764c918"
server_key_version: "encryptionkey@0"
I20260810 18:10:34.163491 11924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:10:34.166340 11941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:34.167819 11924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:10:34.168020 11924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "62b83a4acae34e81bde81abf8ac8466c"
format_stamp: "Formatted at 2026-08-10 18:10:34 on dist-test-slave-27sr"
server_key: "d6c139e70a7765225ff3812c641ea39b"
server_key_iv: "f5998fd8c4c5299924f6debaf764c918"
server_key_version: "encryptionkey@0"
I20260810 18:10:34.168157 11924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:34.199066 11924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:34.202504 11924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:34.202742 11924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:34.203426 11924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:34.204403 11924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:34.204478 11924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:34.204555 11924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:34.204597 11924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:34.214376 11924 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:39277
I20260810 18:10:34.214406 12074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:39277 every 8 connection(s)
I20260810 18:10:34.215395 11924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:10:34.218815 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 11924
I20260810 18:10:34.218991 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:10:34.219347 22233 external_mini_cluster.cc:1428] Setting key fceb13cd205d4f0875d9ab064e3489b1
Aug 10 18:10:34 dist-test-slave-27sr krb5kdc[11468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385434, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.67@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:10:34.231187 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.217491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:34769 (local address 127.21.182.126:35145)
0810 18:10:34.217646 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:34.217649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:34.218555 (+ 906us) server_negotiation.cc:407] Connection header received
0810 18:10:34.220405 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:34.220409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:34.220471 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:34.220555 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:34.222022 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.222622 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.223861 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.224102 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.224186 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.227013 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:34.227017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:34.227019 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:34.227046 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:34.228863 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:34.229449 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:34.229453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:34.229455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:34.229551 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:34.229936 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:34.229940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:34.229941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:34.230158 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:34.230274 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:34.230847 (+ 573us) server_negotiation.cc:299] Negotiation successful
0810 18:10:34.231006 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:10:34.232244 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.217878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:35145 (local address 127.21.182.67:34769)
0810 18:10:34.218364 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:34.218410 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:34.220150 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:34.220772 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:34.220781 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:34.221204 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:34.221823 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:34.221853 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.222752 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.222754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:34.223730 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:34.223741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.224250 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.224253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:34.224404 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.225057 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:10:34.225074 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:34.226815 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:34.229007 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:34.229014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:34.229034 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:34.229335 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:34.229651 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:10:34.229654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:10:34.229656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:10:34.229822 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:10:34.230273 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:34.230281 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:34.230593 (+ 312us) client_negotiation.cc:769] Sending connection context
0810 18:10:34.230804 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 18:10:34.231198 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":120,"threads_started":1}
I20260810 18:10:34.233662 12075 heartbeater.cc:347] Connected to a master server at 127.21.182.126:35145
I20260810 18:10:34.234094 12075 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:34.234760 12075 heartbeater.cc:511] Master 127.21.182.126:35145 requested a full tablet report, sending...
I20260810 18:10:34.236238 11525 ts_manager.cc:194] Registered new tserver with Master: 62b83a4acae34e81bde81abf8ac8466c (127.21.182.67:39277)
I20260810 18:10:34.236843 11525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.182.67@KRBTEST.COM'} at 127.21.182.67:34769
I20260810 18:10:34.246739 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:10:34.257385 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.249072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.21.182.126:35145)
0810 18:10:34.249208 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:34.249212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:34.249404 (+ 192us) server_negotiation.cc:407] Connection header received
0810 18:10:34.249519 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:34.249522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:34.249569 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:34.249642 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:34.250392 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.250988 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.251851 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.252144 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.252240 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.253670 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:34.253674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:34.253688 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:34.253716 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:34.255327 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:34.255829 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:34.255833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:34.255834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:34.256029 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:34.256364 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:34.256368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:34.256369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:34.256748 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:34.256889 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:34.257115 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 18:10:34.257246 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:10:34.260972 11525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:10:34.264353 11525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:10:34.281208 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.275177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:45083 (local address 127.0.0.1:47260)
0810 18:10:34.276077 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:34.276091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:34.276178 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:34.276689 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:34.276696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:34.276735 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:34.276983 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:34.276988 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.278037 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.278044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:34.279363 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:34.279373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.280559 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.280563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:34.280691 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.280736 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 18:10:34.280897 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 18:10:34.281043 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":66,"spinlock_wait_cycles":9088,"thread_start_us":111,"threads_started":1}
I20260810 18:10:34.281212 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.275404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.21.182.66:45083)
0810 18:10:34.275930 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:34.275946 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:34.276267 (+ 321us) server_negotiation.cc:407] Connection header received
0810 18:10:34.276346 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:34.276350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:34.276519 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:34.276657 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:34.277121 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.277895 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.279504 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.280420 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.280545 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.280723 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:34.280928 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 18:10:34.281042 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":132,"threads_started":1}
I20260810 18:10:34.281448 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.274663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49422 (local address 127.21.182.65:39893)
0810 18:10:34.275045 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:34.275056 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:34.275732 (+ 676us) server_negotiation.cc:407] Connection header received
0810 18:10:34.276059 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:34.276066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:34.276311 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:34.276469 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:34.277165 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.278332 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.279605 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.280533 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.280661 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.280841 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:34.281158 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 18:10:34.281263 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":65,"threads_started":1}
I20260810 18:10:34.281952 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.274345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:39893 (local address 127.0.0.1:49422)
0810 18:10:34.275577 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:34.275622 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:34.275831 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:34.276689 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:34.276696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:34.276733 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:34.276978 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:34.276986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.278499 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.278508 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:34.279460 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:34.279475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.280906 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.280910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:34.280995 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.281019 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:10:34.281703 (+ 684us) client_negotiation.cc:240] Negotiation successful
0810 18:10:34.281790 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"mutex_wait_us":83,"spinlock_wait_cycles":7168,"thread_start_us":80,"threads_started":1}
I20260810 18:10:34.282649 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.274345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:39277 (local address 127.0.0.1:46390)
0810 18:10:34.275557 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:34.275619 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:34.275784 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:34.276745 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:34.276749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:34.276766 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:34.276930 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:34.276936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.278459 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.278462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:34.280578 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:34.280592 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.281913 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.281917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:34.282361 (+ 444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.282388 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:10:34.282458 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 18:10:34.282529 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"mutex_wait_us":4,"thread_start_us":116,"threads_started":1}
I20260810 18:10:34.283051 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.274491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46390 (local address 127.21.182.67:39277)
0810 18:10:34.275045 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:34.275061 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:34.276085 (+ 1024us) server_negotiation.cc:407] Connection header received
0810 18:10:34.276182 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:34.276188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:34.276411 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:34.276572 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:34.277060 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.278304 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.280725 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.281594 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.281704 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.281850 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:34.282832 (+ 982us) server_negotiation.cc:299] Negotiation successful
0810 18:10:34.282943 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":194,"threads_started":1}
I20260810 18:10:34.285326 11840 tablet_service.cc:1467] Processing CreateTablet for tablet 437dff329d9342f79d6a7915b0ea70ac (DEFAULT_TABLE table=test-table [id=9e4f910c90ad46cf977bb9844a385f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:34.285621 11683 tablet_service.cc:1467] Processing CreateTablet for tablet 437dff329d9342f79d6a7915b0ea70ac (DEFAULT_TABLE table=test-table [id=9e4f910c90ad46cf977bb9844a385f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:34.285804 11998 tablet_service.cc:1467] Processing CreateTablet for tablet 437dff329d9342f79d6a7915b0ea70ac (DEFAULT_TABLE table=test-table [id=9e4f910c90ad46cf977bb9844a385f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:34.286638 11683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437dff329d9342f79d6a7915b0ea70ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:34.286645 11840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437dff329d9342f79d6a7915b0ea70ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:34.286744 11998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437dff329d9342f79d6a7915b0ea70ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:34.293596 12095 tablet_bootstrap.cc:492] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9: Bootstrap starting.
I20260810 18:10:34.295302 12097 tablet_bootstrap.cc:492] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c: Bootstrap starting.
I20260810 18:10:34.296298 12096 tablet_bootstrap.cc:492] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f: Bootstrap starting.
I20260810 18:10:34.296623 12095 tablet_bootstrap.cc:654] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:34.297689 12097 tablet_bootstrap.cc:654] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:34.297899 12095 log.cc:824] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:34.298584 12096 tablet_bootstrap.cc:654] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:34.299149 12097 log.cc:824] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:34.299886 12096 log.cc:824] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:34.300184 12095 tablet_bootstrap.cc:492] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9: No bootstrap required, opened a new log
I20260810 18:10:34.300432 12095 ts_tablet_manager.cc:1397] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:10:34.301568 12097 tablet_bootstrap.cc:492] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c: No bootstrap required, opened a new log
I20260810 18:10:34.301782 12097 ts_tablet_manager.cc:1397] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 18:10:34.302175 12096 tablet_bootstrap.cc:492] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f: No bootstrap required, opened a new log
I20260810 18:10:34.302421 12096 ts_tablet_manager.cc:1397] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:10:34.304872 12095 raft_consensus.cc:348] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b83a4acae34e81bde81abf8ac8466c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 39277 } } peers { permanent_uuid: "194af3d5682b40f28953bcebd14d45d9" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39893 } } peers { permanent_uuid: "8b0f4d0321d24d29844e14431d364e7f" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 45083 } }
I20260810 18:10:34.305166 12095 raft_consensus.cc:374] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:34.305226 12095 raft_consensus.cc:729] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 194af3d5682b40f28953bcebd14d45d9, State: Initialized, Role: FOLLOWER
I20260810 18:10:34.305739 12095 consensus_queue.cc:260] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b83a4acae34e81bde81abf8ac8466c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 39277 } } peers { permanent_uuid: "194af3d5682b40f28953bcebd14d45d9" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39893 } } peers { permanent_uuid: "8b0f4d0321d24d29844e14431d364e7f" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 45083 } }
I20260810 18:10:34.305946 12097 raft_consensus.cc:348] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b83a4acae34e81bde81abf8ac8466c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 39277 } } peers { permanent_uuid: "194af3d5682b40f28953bcebd14d45d9" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39893 } } peers { permanent_uuid: "8b0f4d0321d24d29844e14431d364e7f" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 45083 } }
I20260810 18:10:34.306221 12097 raft_consensus.cc:374] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:34.306295 12097 raft_consensus.cc:729] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62b83a4acae34e81bde81abf8ac8466c, State: Initialized, Role: FOLLOWER
I20260810 18:10:34.306391 12096 raft_consensus.cc:348] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b83a4acae34e81bde81abf8ac8466c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 39277 } } peers { permanent_uuid: "194af3d5682b40f28953bcebd14d45d9" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39893 } } peers { permanent_uuid: "8b0f4d0321d24d29844e14431d364e7f" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 45083 } }
I20260810 18:10:34.306643 12096 raft_consensus.cc:374] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:34.306641 11756 heartbeater.cc:503] Master 127.21.182.126:35145 was elected leader, sending a full tablet report...
I20260810 18:10:34.306703 12096 raft_consensus.cc:729] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0f4d0321d24d29844e14431d364e7f, State: Initialized, Role: FOLLOWER
I20260810 18:10:34.306833 12097 consensus_queue.cc:260] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b83a4acae34e81bde81abf8ac8466c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 39277 } } peers { permanent_uuid: "194af3d5682b40f28953bcebd14d45d9" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39893 } } peers { permanent_uuid: "8b0f4d0321d24d29844e14431d364e7f" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 45083 } }
I20260810 18:10:34.307123 12096 consensus_queue.cc:260] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b83a4acae34e81bde81abf8ac8466c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 39277 } } peers { permanent_uuid: "194af3d5682b40f28953bcebd14d45d9" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39893 } } peers { permanent_uuid: "8b0f4d0321d24d29844e14431d364e7f" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 45083 } }
I20260810 18:10:34.307610 12075 heartbeater.cc:503] Master 127.21.182.126:35145 was elected leader, sending a full tablet report...
I20260810 18:10:34.307960 12097 ts_tablet_manager.cc:1428] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 18:10:34.308004 12096 ts_tablet_manager.cc:1428] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260810 18:10:34.308318 11919 heartbeater.cc:503] Master 127.21.182.126:35145 was elected leader, sending a full tablet report...
I20260810 18:10:34.308539 12095 ts_tablet_manager.cc:1428] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 18:10:34.395319 12102 raft_consensus.cc:482] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:10:34.395569 12102 raft_consensus.cc:504] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b83a4acae34e81bde81abf8ac8466c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 39277 } } peers { permanent_uuid: "194af3d5682b40f28953bcebd14d45d9" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39893 } } peers { permanent_uuid: "8b0f4d0321d24d29844e14431d364e7f" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 45083 } }
I20260810 18:10:34.397001 12102 leader_election.cc:288] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62b83a4acae34e81bde81abf8ac8466c (127.21.182.67:39277), 8b0f4d0321d24d29844e14431d364e7f (127.21.182.66:45083)
I20260810 18:10:34.402297 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.397399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:39149 (local address 127.21.182.67:39277)
0810 18:10:34.397542 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:34.397546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:34.397738 (+ 192us) server_negotiation.cc:407] Connection header received
0810 18:10:34.397919 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:34.397925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:34.397990 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:34.398084 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:34.398436 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.399248 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.400868 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.401493 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.401623 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.401677 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:34.402088 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 18:10:34.402164 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:10:34.402297 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.398026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:58439 (local address 127.21.182.66:45083)
0810 18:10:34.398165 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:34.398172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:34.398313 (+ 141us) server_negotiation.cc:407] Connection header received
0810 18:10:34.398440 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:34.398442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:34.398495 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:34.398579 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:34.398823 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.399495 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.400868 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.401493 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.401623 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.401677 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:34.402088 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 18:10:34.402164 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:10:34.402374 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.397750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:45083 (local address 127.21.182.65:58439)
0810 18:10:34.398203 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:34.398215 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:34.398321 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:34.398593 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:34.398596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:34.398611 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:34.398716 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:34.398720 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.399597 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.399600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:34.400731 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:34.400738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.401627 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.401631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:34.401882 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.401959 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 18:10:34.402041 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 18:10:34.402149 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":134,"thread_start_us":119,"threads_started":1}
I20260810 18:10:34.402374 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.397303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:39277 (local address 127.21.182.65:39149)
0810 18:10:34.397631 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:34.397649 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:34.397778 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:34.398124 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:34.398127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:34.398147 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:34.398324 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:34.398330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.399364 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.399367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:34.400710 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:34.400723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.401627 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.401631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:34.401876 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.401957 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 18:10:34.402041 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 18:10:34.402149 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":108,"thread_start_us":147,"threads_started":1}
I20260810 18:10:34.403294 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437dff329d9342f79d6a7915b0ea70ac" candidate_uuid: "194af3d5682b40f28953bcebd14d45d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0f4d0321d24d29844e14431d364e7f" is_pre_election: true
I20260810 18:10:34.403290 12024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437dff329d9342f79d6a7915b0ea70ac" candidate_uuid: "194af3d5682b40f28953bcebd14d45d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62b83a4acae34e81bde81abf8ac8466c" is_pre_election: true
I20260810 18:10:34.403633 12024 raft_consensus.cc:2454] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 194af3d5682b40f28953bcebd14d45d9 in term 0.
I20260810 18:10:34.403637 11866 raft_consensus.cc:2454] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 194af3d5682b40f28953bcebd14d45d9 in term 0.
I20260810 18:10:34.404191 11621 leader_election.cc:302] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 194af3d5682b40f28953bcebd14d45d9, 8b0f4d0321d24d29844e14431d364e7f; no voters:
I20260810 18:10:34.404515 12102 raft_consensus.cc:2781] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:10:34.404611 12102 raft_consensus.cc:482] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:10:34.404663 12102 raft_consensus.cc:3037] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:34.405789 12102 raft_consensus.cc:504] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b83a4acae34e81bde81abf8ac8466c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 39277 } } peers { permanent_uuid: "194af3d5682b40f28953bcebd14d45d9" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39893 } } peers { permanent_uuid: "8b0f4d0321d24d29844e14431d364e7f" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 45083 } }
I20260810 18:10:34.406229 12102 leader_election.cc:288] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [CANDIDATE]: Term 1 election: Requested vote from peers 62b83a4acae34e81bde81abf8ac8466c (127.21.182.67:39277), 8b0f4d0321d24d29844e14431d364e7f (127.21.182.66:45083)
I20260810 18:10:34.406620 11866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437dff329d9342f79d6a7915b0ea70ac" candidate_uuid: "194af3d5682b40f28953bcebd14d45d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0f4d0321d24d29844e14431d364e7f"
I20260810 18:10:34.406630 12024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437dff329d9342f79d6a7915b0ea70ac" candidate_uuid: "194af3d5682b40f28953bcebd14d45d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62b83a4acae34e81bde81abf8ac8466c"
I20260810 18:10:34.406807 11866 raft_consensus.cc:3037] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:34.406821 12024 raft_consensus.cc:3037] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:34.408030 12024 raft_consensus.cc:2454] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 194af3d5682b40f28953bcebd14d45d9 in term 1.
I20260810 18:10:34.408115 11866 raft_consensus.cc:2454] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 194af3d5682b40f28953bcebd14d45d9 in term 1.
I20260810 18:10:34.408357 11621 leader_election.cc:302] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 194af3d5682b40f28953bcebd14d45d9, 62b83a4acae34e81bde81abf8ac8466c; no voters:
I20260810 18:10:34.408558 12102 raft_consensus.cc:2781] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:34.408839 12102 raft_consensus.cc:686] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [term 1 LEADER]: Becoming Leader. State: Replica: 194af3d5682b40f28953bcebd14d45d9, State: Running, Role: LEADER
I20260810 18:10:34.409206 12102 consensus_queue.cc:237] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b83a4acae34e81bde81abf8ac8466c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 39277 } } peers { permanent_uuid: "194af3d5682b40f28953bcebd14d45d9" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39893 } } peers { permanent_uuid: "8b0f4d0321d24d29844e14431d364e7f" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 45083 } }
I20260810 18:10:34.412767 11525 catalog_manager.cc:5495] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 194af3d5682b40f28953bcebd14d45d9 (127.21.182.65). New cstate: current_term: 1 leader_uuid: "194af3d5682b40f28953bcebd14d45d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62b83a4acae34e81bde81abf8ac8466c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 39277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "194af3d5682b40f28953bcebd14d45d9" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 39893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b0f4d0321d24d29844e14431d364e7f" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 45083 } health_report { overall_health: UNKNOWN } } }
I20260810 18:10:34.445084 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.440939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49430 (local address 127.21.182.65:39893)
0810 18:10:34.441093 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:34.441096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:34.441110 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:10:34.441209 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:34.441211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:34.441272 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:34.441370 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:10:34.441692 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.442231 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.443161 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.443389 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.443480 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.444023 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:10:34.444523 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:10:34.444636 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:34.444749 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 18:10:34.444819 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:10:34.452255 11866 raft_consensus.cc:1261] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f [term 1 FOLLOWER]: Refusing update from remote peer 194af3d5682b40f28953bcebd14d45d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:10:34.452255 12024 raft_consensus.cc:1261] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c [term 1 FOLLOWER]: Refusing update from remote peer 194af3d5682b40f28953bcebd14d45d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:10:34.453097 12102 consensus_queue.cc:1035] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62b83a4acae34e81bde81abf8ac8466c" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 39277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:10:34.453351 12108 consensus_queue.cc:1035] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b0f4d0321d24d29844e14431d364e7f" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 45083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:10:34.464247 12111 mvcc.cc:205] Tried to move back new op lower bound from 7317034739507032064 to 7317034739343036416. Current Snapshot: MvccSnapshot[applied={T|T < 7317034739507032064}]
I20260810 18:10:34.464622 12112 mvcc.cc:205] Tried to move back new op lower bound from 7317034739507032064 to 7317034739343036416. Current Snapshot: MvccSnapshot[applied={T|T < 7317034739507032064}]
I20260810 18:10:34.473732 11523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:10:34.473924 11523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:10:34.477231 11523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82d872f71976405f955f6cdeb633169a: Sending DeleteTablet for 3 replicas of tablet 437dff329d9342f79d6a7915b0ea70ac
I20260810 18:10:34.478315 11998 tablet_service.cc:1514] Processing DeleteTablet for tablet 437dff329d9342f79d6a7915b0ea70ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:34 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:46390
I20260810 18:10:34.478327 11840 tablet_service.cc:1514] Processing DeleteTablet for tablet 437dff329d9342f79d6a7915b0ea70ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:34 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:47260
I20260810 18:10:34.478978 12123 tablet_replica.cc:330] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c: stopping tablet replica
I20260810 18:10:34.479003 11683 tablet_service.cc:1514] Processing DeleteTablet for tablet 437dff329d9342f79d6a7915b0ea70ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:34 UTC) from {username='kudu', principal='kudu/127.21.182.126@KRBTEST.COM'} at 127.0.0.1:49422
I20260810 18:10:34.479439 12124 tablet_replica.cc:330] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f: stopping tablet replica
I20260810 18:10:34.479573 12126 tablet_replica.cc:330] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9: stopping tablet replica
I20260810 18:10:34.479715 12123 raft_consensus.cc:2229] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:10:34.479759 12124 raft_consensus.cc:2229] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:10:34.479910 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 11600
I20260810 18:10:34.479947 12126 raft_consensus.cc:2229] T 437dff329d9342f79d6a7915b0ea70ac P 194af3d5682b40f28953bcebd14d45d9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:10:34.481530 12124 raft_consensus.cc:2258] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:34.483963 12124 ts_tablet_manager.cc:1905] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:10:34.486308 12123 raft_consensus.cc:2258] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:34.487752 12124 ts_tablet_manager.cc:1918] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:10:34.487881 12124 log.cc:1196] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/437dff329d9342f79d6a7915b0ea70ac
I20260810 18:10:34.488303 12124 ts_tablet_manager.cc:1939] T 437dff329d9342f79d6a7915b0ea70ac P 8b0f4d0321d24d29844e14431d364e7f: Deleting consensus metadata
I20260810 18:10:34.488770 12123 ts_tablet_manager.cc:1905] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:10:34.489966 11508 catalog_manager.cc:4841] TS 8b0f4d0321d24d29844e14431d364e7f (127.21.182.66:45083): tablet 437dff329d9342f79d6a7915b0ea70ac (table test-table [id=9e4f910c90ad46cf977bb9844a385f42]) successfully deleted
I20260810 18:10:34.492672 12123 ts_tablet_manager.cc:1918] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:10:34.492799 12123 log.cc:1196] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/437dff329d9342f79d6a7915b0ea70ac
I20260810 18:10:34.493202 12123 ts_tablet_manager.cc:1939] T 437dff329d9342f79d6a7915b0ea70ac P 62b83a4acae34e81bde81abf8ac8466c: Deleting consensus metadata
I20260810 18:10:34.494760 11508 catalog_manager.cc:4841] TS 62b83a4acae34e81bde81abf8ac8466c (127.21.182.67:39277): tablet 437dff329d9342f79d6a7915b0ea70ac (table test-table [id=9e4f910c90ad46cf977bb9844a385f42]) successfully deleted
W20260810 18:10:34.498934 11510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:39893 (error 108)
I20260810 18:10:34.499089 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 11763
I20260810 18:10:34.504042 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.503530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:39893 (local address 127.0.0.1:49438)
0810 18:10:34.503775 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:34.503861 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:39893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260810 18:10:34.504288 11510 catalog_manager.cc:4584] TS 194af3d5682b40f28953bcebd14d45d9 (127.21.182.65:39893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 437dff329d9342f79d6a7915b0ea70ac: Network error: Client connection negotiation failed: client connection to 127.21.182.65:39893: connect: Connection refused (error 111)
I20260810 18:10:34.510159 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 11924
I20260810 18:10:34.518723 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 11488
2026-08-10T18:10:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2622 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T18:10:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:34Z Disabled control of system clock
I20260810 18:10:34.555421 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:43947
--webserver_interface=127.21.182.126
--webserver_port=0
--builtin_ntp_servers=127.21.182.84:43357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:43947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:10:34.678920 12132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:34.679246 12132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:34.679286 12132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:34.683606 12132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:34.683696 12132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:34.683712 12132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:34.683730 12132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:34.683745 12132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:34.688867 12132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:43357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:43947
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:43947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:34.690075 12132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:34.691705 12132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:34.701349 12141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:34.701514 12132 server_base.cc:1034] running on GCE node
W20260810 18:10:34.701349 12139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:34.701835 12138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:34.702448 12132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:34.703688 12132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:34.704854 12132 hybrid_clock.cc:648] HybridClock initialized: now 1786385434704816 us; error 58 us; skew 500 ppm
I20260810 18:10:34.706965 12132 webserver.cc:460] Webserver started at http://127.21.182.126:43061/ using document root <none> and password file <none>
I20260810 18:10:34.707624 12132 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:34.707692 12132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:34.708003 12132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:34.710587 12132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "6a954a9c8f434af898182abbfa81c4f9"
format_stamp: "Formatted at 2026-08-10 18:10:34 on dist-test-slave-27sr"
server_key: "0c46edac590e6eabf20acdb34f44653d"
server_key_iv: "80e317e7a56d168d5ff8b2254b828d99"
server_key_version: "encryptionkey@0"
I20260810 18:10:34.711261 12132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "6a954a9c8f434af898182abbfa81c4f9"
format_stamp: "Formatted at 2026-08-10 18:10:34 on dist-test-slave-27sr"
server_key: "0c46edac590e6eabf20acdb34f44653d"
server_key_iv: "80e317e7a56d168d5ff8b2254b828d99"
server_key_version: "encryptionkey@0"
I20260810 18:10:34.715482 12132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:10:34.718382 12147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:34.720031 12132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:34.720197 12132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "6a954a9c8f434af898182abbfa81c4f9"
format_stamp: "Formatted at 2026-08-10 18:10:34 on dist-test-slave-27sr"
server_key: "0c46edac590e6eabf20acdb34f44653d"
server_key_iv: "80e317e7a56d168d5ff8b2254b828d99"
server_key_version: "encryptionkey@0"
I20260810 18:10:34.720333 12132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:34.786808 12132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:34.787662 12132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:34.787956 12132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:34.795697 12132 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:43947
I20260810 18:10:34.795706 12214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:43947 every 8 connection(s)
I20260810 18:10:34.796829 12132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:34.800428 12215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:34.804049 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 12132
I20260810 18:10:34.804196 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:34.804558 22233 external_mini_cluster.cc:1428] Setting key 266cc78673244481d820e799656e4f17
I20260810 18:10:34.807343 12215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9: Bootstrap starting.
I20260810 18:10:34.809870 12215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:34.810698 12215 log.cc:824] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:34.813059 12215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9: No bootstrap required, opened a new log
I20260810 18:10:34.815027 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:34.806256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.21.182.126:43947)
0810 18:10:34.807046 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:34.807063 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:34.807105 (+ 42us) server_negotiation.cc:407] Connection header received
0810 18:10:34.807860 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:34.807881 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:34.808429 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:34.808785 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:34.809268 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.810379 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.811363 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:34.811858 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:34.812099 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:34.813089 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:34.813094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:34.813109 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:34.813596 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:34.813662 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:34.813685 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:34.813801 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:34.813972 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 18:10:34.814257 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":161,"threads_started":1}
I20260810 18:10:34.817365 12215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a954a9c8f434af898182abbfa81c4f9" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43947 } }
I20260810 18:10:34.817692 12215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:34.817803 12215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a954a9c8f434af898182abbfa81c4f9, State: Initialized, Role: FOLLOWER
I20260810 18:10:34.818328 12215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a954a9c8f434af898182abbfa81c4f9" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43947 } }
I20260810 18:10:34.818499 12215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:34.818593 12215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:34.818727 12215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:34.819777 12215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a954a9c8f434af898182abbfa81c4f9" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43947 } }
I20260810 18:10:34.820194 12215 leader_election.cc:302] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a954a9c8f434af898182abbfa81c4f9; no voters:
I20260810 18:10:34.820597 12215 leader_election.cc:288] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:34.820838 12221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:34.821146 12221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a954a9c8f434af898182abbfa81c4f9, State: Running, Role: LEADER
I20260810 18:10:34.821516 12221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a954a9c8f434af898182abbfa81c4f9" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43947 } }
I20260810 18:10:34.822240 12215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:34.823581 12222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a954a9c8f434af898182abbfa81c4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a954a9c8f434af898182abbfa81c4f9" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43947 } } }
I20260810 18:10:34.823729 12222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:34.824152 12226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a954a9c8f434af898182abbfa81c4f9. Latest consensus state: current_term: 1 leader_uuid: "6a954a9c8f434af898182abbfa81c4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a954a9c8f434af898182abbfa81c4f9" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43947 } } }
I20260810 18:10:34.824277 12226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:34.824605 12235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:34.828723 12235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:34.835269 12235 catalog_manager.cc:1347] Generated new cluster ID: 815b7e8b3cbd4774afc082b2c86fc995
I20260810 18:10:34.835371 12235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:34.851351 12235 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:34.852591 12235 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:34.860705 12235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9: Generated new TSK 0
I20260810 18:10:34.861723 12235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:34.867501 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.65:0
--local_ip_for_outbound_sockets=127.21.182.65
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43947
--builtin_ntp_servers=127.21.182.84:43357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 18:10:34.994450 12246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:34.994753 12246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:34.994828 12246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:34.998917 12246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:34.999003 12246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:34.999096 12246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.65
I20260810 18:10:35.003949 12246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:43357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.182.65
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.65
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:35.005090 12246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:35.006528 12246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:35.016507 12253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:35.016592 12252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:35.016554 12246 server_base.cc:1034] running on GCE node
W20260810 18:10:35.016558 12255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:35.017200 12246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:35.017855 12246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:35.019047 12246 hybrid_clock.cc:648] HybridClock initialized: now 1786385435019014 us; error 44 us; skew 500 ppm
I20260810 18:10:35.020958 12246 webserver.cc:460] Webserver started at http://127.21.182.65:36897/ using document root <none> and password file <none>
I20260810 18:10:35.021559 12246 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:35.021656 12246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:35.021894 12246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:35.024199 12246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/data/instance:
uuid: "006c39d9d868403d9ee52c94a7f4670a"
format_stamp: "Formatted at 2026-08-10 18:10:35 on dist-test-slave-27sr"
server_key: "72b5998d8bcca56e223872527b4026c8"
server_key_iv: "e93d1d3c30f1d625dd47443f407809a5"
server_key_version: "encryptionkey@0"
I20260810 18:10:35.024875 12246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance:
uuid: "006c39d9d868403d9ee52c94a7f4670a"
format_stamp: "Formatted at 2026-08-10 18:10:35 on dist-test-slave-27sr"
server_key: "72b5998d8bcca56e223872527b4026c8"
server_key_iv: "e93d1d3c30f1d625dd47443f407809a5"
server_key_version: "encryptionkey@0"
I20260810 18:10:35.028705 12246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:10:35.031147 12262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:35.032491 12246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:10:35.032645 12246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/wal
uuid: "006c39d9d868403d9ee52c94a7f4670a"
format_stamp: "Formatted at 2026-08-10 18:10:35 on dist-test-slave-27sr"
server_key: "72b5998d8bcca56e223872527b4026c8"
server_key_iv: "e93d1d3c30f1d625dd47443f407809a5"
server_key_version: "encryptionkey@0"
I20260810 18:10:35.032768 12246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:35.053989 12246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:35.054812 12246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:35.055040 12246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:35.055707 12246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:35.056792 12246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:35.056870 12246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:35.056947 12246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:35.056993 12246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:35.066938 12246 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.65:41799
I20260810 18:10:35.066960 12400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.65:41799 every 8 connection(s)
I20260810 18:10:35.068030 12246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/data/info.pb
I20260810 18:10:35.076097 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 12246
I20260810 18:10:35.076223 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-0/wal/instance
I20260810 18:10:35.076587 22233 external_mini_cluster.cc:1428] Setting key 589fb3a7a1e68f4408125878516a0ce2
I20260810 18:10:35.078727 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.070240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.65:55073 (local address 127.21.182.126:43947)
0810 18:10:35.070397 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:35.070401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:35.071587 (+ 1186us) server_negotiation.cc:407] Connection header received
0810 18:10:35.073277 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:35.073280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:35.073352 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:35.073458 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:35.074448 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.075019 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.076510 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.076806 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.076925 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.078171 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:35.078176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:35.078178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:35.078201 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:35.078230 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:35.078239 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:35.078319 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:35.078495 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 18:10:35.078561 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:10:35.078748 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.66:0
--local_ip_for_outbound_sockets=127.21.182.66
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43947
--builtin_ntp_servers=127.21.182.84:43357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:10:35.079699 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.070667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:43947 (local address 127.21.182.65:55073)
0810 18:10:35.071408 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:35.071453 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:35.073014 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:35.073660 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:35.073667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:35.073767 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:35.074289 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.074303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.075161 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.075170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.076360 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.076371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.076980 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.076984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.077168 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.077878 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:10:35.077880 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:35.077926 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:10:35.078003 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:35.078324 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:35.078337 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:35.078345 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 18:10:35.078494 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 18:10:35.078856 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":134,"threads_started":1}
I20260810 18:10:35.081171 12401 heartbeater.cc:347] Connected to a master server at 127.21.182.126:43947
I20260810 18:10:35.081503 12401 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:35.082100 12401 heartbeater.cc:511] Master 127.21.182.126:43947 requested a full tablet report, sending...
I20260810 18:10:35.084141 12169 ts_manager.cc:194] Registered new tserver with Master: 006c39d9d868403d9ee52c94a7f4670a (127.21.182.65:41799)
I20260810 18:10:35.085491 12169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.182.65:55073
W20260810 18:10:35.203143 12405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:35.203502 12405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:35.203579 12405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:35.208078 12405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:35.208181 12405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:35.208282 12405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.66
I20260810 18:10:35.213533 12405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:43357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.182.66
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.66
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:35.214778 12405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:35.216336 12405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:35.227237 12413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:35.227213 12410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:35.227341 12405 server_base.cc:1034] running on GCE node
W20260810 18:10:35.227196 12411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:35.228024 12405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:35.228783 12405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:35.229974 12405 hybrid_clock.cc:648] HybridClock initialized: now 1786385435229951 us; error 34 us; skew 500 ppm
I20260810 18:10:35.232115 12405 webserver.cc:460] Webserver started at http://127.21.182.66:35893/ using document root <none> and password file <none>
I20260810 18:10:35.232792 12405 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:35.232931 12405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:35.233189 12405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:35.235508 12405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/data/instance:
uuid: "35587c576db64103947f236cb6d071a1"
format_stamp: "Formatted at 2026-08-10 18:10:35 on dist-test-slave-27sr"
server_key: "1a945995c01093b015a090f6cdac98cc"
server_key_iv: "a7c8bcea4820a1b7d9f13ea14c6e0887"
server_key_version: "encryptionkey@0"
I20260810 18:10:35.236241 12405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance:
uuid: "35587c576db64103947f236cb6d071a1"
format_stamp: "Formatted at 2026-08-10 18:10:35 on dist-test-slave-27sr"
server_key: "1a945995c01093b015a090f6cdac98cc"
server_key_iv: "a7c8bcea4820a1b7d9f13ea14c6e0887"
server_key_version: "encryptionkey@0"
I20260810 18:10:35.240964 12405 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 18:10:35.243842 12420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:35.245360 12405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:10:35.245532 12405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/wal
uuid: "35587c576db64103947f236cb6d071a1"
format_stamp: "Formatted at 2026-08-10 18:10:35 on dist-test-slave-27sr"
server_key: "1a945995c01093b015a090f6cdac98cc"
server_key_iv: "a7c8bcea4820a1b7d9f13ea14c6e0887"
server_key_version: "encryptionkey@0"
I20260810 18:10:35.245672 12405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:35.266922 12405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:35.267832 12405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:35.268090 12405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:35.268760 12405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:35.269848 12405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:35.269933 12405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:35.270010 12405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:35.270061 12405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:35.280221 12405 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.66:32935
I20260810 18:10:35.280242 12547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.66:32935 every 8 connection(s)
I20260810 18:10:35.281273 12405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/data/info.pb
I20260810 18:10:35.288348 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 12405
I20260810 18:10:35.288539 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/wal/instance
I20260810 18:10:35.288905 22233 external_mini_cluster.cc:1428] Setting key 30be73bfea3ab99a3f8abadce786b2e6
I20260810 18:10:35.291554 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.182.67:0
--local_ip_for_outbound_sockets=127.21.182.67
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43947
--builtin_ntp_servers=127.21.182.84:43357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 18:10:35.292133 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.283361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:39143 (local address 127.21.182.126:43947)
0810 18:10:35.283523 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:35.283527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:35.284758 (+ 1231us) server_negotiation.cc:407] Connection header received
0810 18:10:35.286214 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:35.286216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:35.286271 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:35.286361 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:35.287395 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.288042 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.289790 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.290089 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.290204 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.291541 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:35.291544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:35.291546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:35.291559 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:35.291581 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:35.291589 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:35.291694 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:35.291922 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 18:10:35.291991 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:10:35.293140 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.283814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:43947 (local address 127.21.182.66:39143)
0810 18:10:35.284586 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:35.284633 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:35.285986 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:35.286571 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:35.286579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:35.286674 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:35.287225 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.287239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.288199 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.288202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.289643 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.289658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.290256 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.290259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.290468 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.291191 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:10:35.291193 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:35.291261 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:10:35.291362 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:35.291697 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:35.291711 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:35.291721 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 18:10:35.291888 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 18:10:35.292248 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":131,"threads_started":1}
I20260810 18:10:35.294499 12548 heartbeater.cc:347] Connected to a master server at 127.21.182.126:43947
I20260810 18:10:35.294763 12548 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:35.295394 12548 heartbeater.cc:511] Master 127.21.182.126:43947 requested a full tablet report, sending...
I20260810 18:10:35.296656 12169 ts_manager.cc:194] Registered new tserver with Master: 35587c576db64103947f236cb6d071a1 (127.21.182.66:32935)
I20260810 18:10:35.297300 12169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.182.66:39143
W20260810 18:10:35.413518 12551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:35.413802 12551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:35.413874 12551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:35.417979 12551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:35.418066 12551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:35.418154 12551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.182.67
I20260810 18:10:35.424201 12551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:43357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.182.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.182.67
--webserver_port=0
--tserver_master_addrs=127.21.182.126:43947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.182.67
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:35.425431 12551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:35.426863 12551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:35.437368 12556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:35.437393 12557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:35.437368 12559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:35.437978 12551 server_base.cc:1034] running on GCE node
I20260810 18:10:35.438458 12551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:35.439199 12551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:35.440407 12551 hybrid_clock.cc:648] HybridClock initialized: now 1786385435440368 us; error 49 us; skew 500 ppm
I20260810 18:10:35.442619 12551 webserver.cc:460] Webserver started at http://127.21.182.67:33241/ using document root <none> and password file <none>
I20260810 18:10:35.443367 12551 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:35.443475 12551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:35.443750 12551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:35.446293 12551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/data/instance:
uuid: "bf1f19591de147bd9dee03d0b2c292b7"
format_stamp: "Formatted at 2026-08-10 18:10:35 on dist-test-slave-27sr"
server_key: "c28782e2fc4e660f83f2e5b2ca07b111"
server_key_iv: "bf835e8e7a9622e8bf63755f8e35b997"
server_key_version: "encryptionkey@0"
I20260810 18:10:35.446987 12551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance:
uuid: "bf1f19591de147bd9dee03d0b2c292b7"
format_stamp: "Formatted at 2026-08-10 18:10:35 on dist-test-slave-27sr"
server_key: "c28782e2fc4e660f83f2e5b2ca07b111"
server_key_iv: "bf835e8e7a9622e8bf63755f8e35b997"
server_key_version: "encryptionkey@0"
I20260810 18:10:35.451824 12551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 18:10:35.454672 12566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:35.456226 12551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:35.456377 12551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/wal
uuid: "bf1f19591de147bd9dee03d0b2c292b7"
format_stamp: "Formatted at 2026-08-10 18:10:35 on dist-test-slave-27sr"
server_key: "c28782e2fc4e660f83f2e5b2ca07b111"
server_key_iv: "bf835e8e7a9622e8bf63755f8e35b997"
server_key_version: "encryptionkey@0"
I20260810 18:10:35.456516 12551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:35.475499 12551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:35.476362 12551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:35.476648 12551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:35.477301 12551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:10:35.478354 12551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:10:35.478439 12551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:35.478516 12551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:10:35.478569 12551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:35.488304 12551 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.67:45801
I20260810 18:10:35.488330 12695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.67:45801 every 8 connection(s)
I20260810 18:10:35.489290 12551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/data/info.pb
I20260810 18:10:35.490358 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 12551
I20260810 18:10:35.490507 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/wal/instance
I20260810 18:10:35.490810 22233 external_mini_cluster.cc:1428] Setting key e8ada8c8d6644c25a9d8cf98e02d9b3b
I20260810 18:10:35.500046 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.491656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.67:35701 (local address 127.21.182.126:43947)
0810 18:10:35.491815 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:35.491819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:35.493405 (+ 1586us) server_negotiation.cc:407] Connection header received
0810 18:10:35.494886 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:35.494889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:35.494949 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:35.495046 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:35.496010 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.496553 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.497746 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.497975 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.498093 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.499386 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:35.499390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:35.499393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:35.499425 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:35.499448 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:35.499461 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:35.499531 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:35.499790 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 18:10:35.499858 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:10:35.501060 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.492247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:43947 (local address 127.21.182.67:35701)
0810 18:10:35.493250 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:35.493299 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:35.494641 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:35.495228 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:35.495234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:35.495319 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:10:35.495816 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.495829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.496709 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.496712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.497604 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.497614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.498136 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.498140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.498318 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.499043 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:10:35.499046 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:10:35.499112 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:10:35.499223 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:10:35.499581 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:10:35.499595 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:10:35.499604 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:10:35.499792 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 18:10:35.500272 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":189,"threads_started":1}
I20260810 18:10:35.502205 12697 heartbeater.cc:347] Connected to a master server at 127.21.182.126:43947
I20260810 18:10:35.502470 12697 heartbeater.cc:464] Registering TS with master...
I20260810 18:10:35.503026 12697 heartbeater.cc:511] Master 127.21.182.126:43947 requested a full tablet report, sending...
I20260810 18:10:35.504323 12169 ts_manager.cc:194] Registered new tserver with Master: bf1f19591de147bd9dee03d0b2c292b7 (127.21.182.67:45801)
I20260810 18:10:35.505100 12169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.182.67:35701
I20260810 18:10:35.505465 22233 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:10:35.511683 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.507605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.21.182.126:43947)
0810 18:10:35.507766 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:35.507773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:35.508046 (+ 273us) server_negotiation.cc:407] Connection header received
0810 18:10:35.508183 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:35.508186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:35.508236 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:35.508319 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:35.508617 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.509140 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.510011 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.510233 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.510342 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.511240 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:35.511244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:35.511247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:35.511260 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:35.511287 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:35.511295 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:35.511372 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:35.511473 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 18:10:35.511528 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 18:10:35.514830 12169 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:10:35.518132 12169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:10:35.534538 12715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.528107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:41799 (local address 127.0.0.1:56334)
0810 18:10:35.528748 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:35.528764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:35.528854 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:35.529965 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:35.529968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:35.529986 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:35.530087 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.530095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.531250 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.531255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.532773 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.532780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.533926 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.533930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.534045 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.534088 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 18:10:35.534216 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 18:10:35.534342 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":106,"thread_start_us":114,"threads_started":1}
I20260810 18:10:35.534571 12716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.528579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.21.182.65:41799)
0810 18:10:35.529115 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:35.529129 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:35.529156 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:10:35.529240 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:35.529246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:35.529455 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:35.529994 (+ 539us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:35.530188 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.531131 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.533000 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.533796 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.533894 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.534042 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:35.534270 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 18:10:35.534404 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":148,"threads_started":1}
I20260810 18:10:35.535127 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.528470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.66:32935 (local address 127.0.0.1:44780)
0810 18:10:35.529696 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:35.529707 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:35.529817 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:35.530266 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:35.530269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:35.530288 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:35.530363 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.530368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.532030 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.532034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.533493 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.533505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.534700 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.534705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.534800 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.534839 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 18:10:35.534925 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 18:10:35.534988 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"thread_start_us":106,"threads_started":1}
I20260810 18:10:35.535634 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.529272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44780 (local address 127.21.182.66:32935)
0810 18:10:35.529661 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:35.529674 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:35.529830 (+ 156us) server_negotiation.cc:407] Connection header received
0810 18:10:35.529917 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:35.529922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:35.530123 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:35.530288 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:35.530460 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.531861 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.533631 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.534349 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.534451 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.534632 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:35.535384 (+ 752us) server_negotiation.cc:299] Negotiation successful
0810 18:10:35.535480 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":127,"threads_started":1}
I20260810 18:10:35.535835 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.527703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:45801 (local address 127.0.0.1:56084)
0810 18:10:35.528519 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:35.528562 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:35.528761 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:35.529667 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:35.529675 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:35.529716 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:35.529896 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.529904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.531515 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.531519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.532754 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.532766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.535408 (+ 2642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.535412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.535511 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.535534 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:10:35.535619 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 18:10:35.535687 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":42,"thread_start_us":90,"threads_started":1}
I20260810 18:10:35.536260 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.527932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.21.182.67:45801)
0810 18:10:35.528430 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:35.528444 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:35.528938 (+ 494us) server_negotiation.cc:407] Connection header received
0810 18:10:35.529036 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:35.529040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:35.529282 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:35.529450 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:35.530055 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.531393 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.534097 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.535066 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.535174 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.535339 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:35.535677 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 18:10:35.536105 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":132,"threads_started":1}
I20260810 18:10:35.538571 12476 tablet_service.cc:1467] Processing CreateTablet for tablet 2f402d83589b454c851fd45bdc15439a (DEFAULT_TABLE table=test-table [id=0a31a376a4994ef4b4ad169de9eaadc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:35.538904 12319 tablet_service.cc:1467] Processing CreateTablet for tablet 2f402d83589b454c851fd45bdc15439a (DEFAULT_TABLE table=test-table [id=0a31a376a4994ef4b4ad169de9eaadc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:35.539222 12622 tablet_service.cc:1467] Processing CreateTablet for tablet 2f402d83589b454c851fd45bdc15439a (DEFAULT_TABLE table=test-table [id=0a31a376a4994ef4b4ad169de9eaadc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:10:35.539726 12476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f402d83589b454c851fd45bdc15439a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:35.540027 12319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f402d83589b454c851fd45bdc15439a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:35.540482 12622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f402d83589b454c851fd45bdc15439a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:35.546433 12718 tablet_bootstrap.cc:492] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1: Bootstrap starting.
I20260810 18:10:35.548055 12720 tablet_bootstrap.cc:492] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7: Bootstrap starting.
I20260810 18:10:35.548645 12719 tablet_bootstrap.cc:492] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a: Bootstrap starting.
I20260810 18:10:35.548978 12718 tablet_bootstrap.cc:654] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:35.549810 12718 log.cc:824] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:35.550319 12720 tablet_bootstrap.cc:654] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:35.550997 12719 tablet_bootstrap.cc:654] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:35.551215 12720 log.cc:824] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:35.551843 12719 log.cc:824] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:35.552172 12718 tablet_bootstrap.cc:492] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1: No bootstrap required, opened a new log
I20260810 18:10:35.552392 12718 ts_tablet_manager.cc:1397] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:10:35.553367 12720 tablet_bootstrap.cc:492] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7: No bootstrap required, opened a new log
I20260810 18:10:35.553561 12720 ts_tablet_manager.cc:1397] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:10:35.553738 12719 tablet_bootstrap.cc:492] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a: No bootstrap required, opened a new log
I20260810 18:10:35.553936 12719 ts_tablet_manager.cc:1397] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:10:35.556960 12718 raft_consensus.cc:348] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f19591de147bd9dee03d0b2c292b7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45801 } } peers { permanent_uuid: "35587c576db64103947f236cb6d071a1" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 32935 } } peers { permanent_uuid: "006c39d9d868403d9ee52c94a7f4670a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 41799 } }
I20260810 18:10:35.557260 12718 raft_consensus.cc:374] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:35.557323 12718 raft_consensus.cc:729] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35587c576db64103947f236cb6d071a1, State: Initialized, Role: FOLLOWER
I20260810 18:10:35.557142 12719 raft_consensus.cc:348] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f19591de147bd9dee03d0b2c292b7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45801 } } peers { permanent_uuid: "35587c576db64103947f236cb6d071a1" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 32935 } } peers { permanent_uuid: "006c39d9d868403d9ee52c94a7f4670a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 41799 } }
I20260810 18:10:35.557394 12719 raft_consensus.cc:374] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:35.557463 12719 raft_consensus.cc:729] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006c39d9d868403d9ee52c94a7f4670a, State: Initialized, Role: FOLLOWER
I20260810 18:10:35.557840 12718 consensus_queue.cc:260] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f19591de147bd9dee03d0b2c292b7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45801 } } peers { permanent_uuid: "35587c576db64103947f236cb6d071a1" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 32935 } } peers { permanent_uuid: "006c39d9d868403d9ee52c94a7f4670a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 41799 } }
I20260810 18:10:35.557889 12719 consensus_queue.cc:260] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f19591de147bd9dee03d0b2c292b7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45801 } } peers { permanent_uuid: "35587c576db64103947f236cb6d071a1" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 32935 } } peers { permanent_uuid: "006c39d9d868403d9ee52c94a7f4670a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 41799 } }
I20260810 18:10:35.558612 12401 heartbeater.cc:503] Master 127.21.182.126:43947 was elected leader, sending a full tablet report...
I20260810 18:10:35.558974 12719 ts_tablet_manager.cc:1428] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:10:35.558974 12718 ts_tablet_manager.cc:1428] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:10:35.558779 12720 raft_consensus.cc:348] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f19591de147bd9dee03d0b2c292b7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45801 } } peers { permanent_uuid: "35587c576db64103947f236cb6d071a1" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 32935 } } peers { permanent_uuid: "006c39d9d868403d9ee52c94a7f4670a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 41799 } }
I20260810 18:10:35.559046 12720 raft_consensus.cc:374] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:35.559093 12720 raft_consensus.cc:729] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1f19591de147bd9dee03d0b2c292b7, State: Initialized, Role: FOLLOWER
I20260810 18:10:35.559327 12548 heartbeater.cc:503] Master 127.21.182.126:43947 was elected leader, sending a full tablet report...
I20260810 18:10:35.559509 12720 consensus_queue.cc:260] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f19591de147bd9dee03d0b2c292b7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45801 } } peers { permanent_uuid: "35587c576db64103947f236cb6d071a1" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 32935 } } peers { permanent_uuid: "006c39d9d868403d9ee52c94a7f4670a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 41799 } }
I20260810 18:10:35.560556 12720 ts_tablet_manager.cc:1428] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 18:10:35.560719 12697 heartbeater.cc:503] Master 127.21.182.126:43947 was elected leader, sending a full tablet report...
W20260810 18:10:35.569243 12402 tablet.cc:2369] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:10:35.608186 12726 raft_consensus.cc:482] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:10:35.608434 12726 raft_consensus.cc:504] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f19591de147bd9dee03d0b2c292b7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45801 } } peers { permanent_uuid: "35587c576db64103947f236cb6d071a1" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 32935 } } peers { permanent_uuid: "006c39d9d868403d9ee52c94a7f4670a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 41799 } }
I20260810 18:10:35.609733 12726 leader_election.cc:288] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf1f19591de147bd9dee03d0b2c292b7 (127.21.182.67:45801), 006c39d9d868403d9ee52c94a7f4670a (127.21.182.65:41799)
I20260810 18:10:35.613456 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.609937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.67:45801 (local address 127.21.182.66:49533)
0810 18:10:35.610092 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:35.610111 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:35.610218 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:35.610519 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:35.610522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:35.610541 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:35.610663 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.610669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.611442 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.611446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.612361 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.612371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.613019 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.613022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.613129 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.613147 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:10:35.613239 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 18:10:35.613313 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":33}
I20260810 18:10:35.613454 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.610078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:49533 (local address 127.21.182.67:45801)
0810 18:10:35.610221 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:35.610226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:35.610243 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:10:35.610321 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:35.610324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:35.610389 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:35.610502 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:35.610779 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.611285 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.612494 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.612889 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.613021 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.613072 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:35.613257 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 18:10:35.613321 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:10:35.614176 12728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.610133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:41799 (local address 127.21.182.66:52325)
0810 18:10:35.610516 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:35.610529 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:10:35.610658 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:10:35.610967 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:10:35.610970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:10:35.610987 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:10:35.611123 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.611130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.611979 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.611984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.612884 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:10:35.612895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.613733 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.613737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:10:35.613830 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.613859 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 18:10:35.613963 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 18:10:35.614035 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":15,"thread_start_us":147,"threads_started":1}
I20260810 18:10:35.614164 12716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.610238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.182.66:52325 (local address 127.21.182.65:41799)
0810 18:10:35.610355 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:35.610359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:35.610633 (+ 274us) server_negotiation.cc:407] Connection header received
0810 18:10:35.610798 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:35.610807 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:35.610863 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:35.610957 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:10:35.611255 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.611815 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.613034 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.613576 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.613724 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.613775 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:35.613985 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 18:10:35.614060 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:10:35.614197 12644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f402d83589b454c851fd45bdc15439a" candidate_uuid: "35587c576db64103947f236cb6d071a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1f19591de147bd9dee03d0b2c292b7" is_pre_election: true
I20260810 18:10:35.614599 12644 raft_consensus.cc:2454] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35587c576db64103947f236cb6d071a1 in term 0.
I20260810 18:10:35.614766 12346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f402d83589b454c851fd45bdc15439a" candidate_uuid: "35587c576db64103947f236cb6d071a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "006c39d9d868403d9ee52c94a7f4670a" is_pre_election: true
I20260810 18:10:35.615095 12346 raft_consensus.cc:2454] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35587c576db64103947f236cb6d071a1 in term 0.
I20260810 18:10:35.615206 12423 leader_election.cc:302] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35587c576db64103947f236cb6d071a1, bf1f19591de147bd9dee03d0b2c292b7; no voters:
I20260810 18:10:35.615587 12726 raft_consensus.cc:2781] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:10:35.615705 12726 raft_consensus.cc:482] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:10:35.615777 12726 raft_consensus.cc:3037] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:35.617091 12726 raft_consensus.cc:504] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f19591de147bd9dee03d0b2c292b7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45801 } } peers { permanent_uuid: "35587c576db64103947f236cb6d071a1" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 32935 } } peers { permanent_uuid: "006c39d9d868403d9ee52c94a7f4670a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 41799 } }
I20260810 18:10:35.617473 12726 leader_election.cc:288] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [CANDIDATE]: Term 1 election: Requested vote from peers bf1f19591de147bd9dee03d0b2c292b7 (127.21.182.67:45801), 006c39d9d868403d9ee52c94a7f4670a (127.21.182.65:41799)
I20260810 18:10:35.617865 12346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f402d83589b454c851fd45bdc15439a" candidate_uuid: "35587c576db64103947f236cb6d071a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "006c39d9d868403d9ee52c94a7f4670a"
I20260810 18:10:35.617887 12644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f402d83589b454c851fd45bdc15439a" candidate_uuid: "35587c576db64103947f236cb6d071a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1f19591de147bd9dee03d0b2c292b7"
I20260810 18:10:35.618006 12644 raft_consensus.cc:3037] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:35.618006 12346 raft_consensus.cc:3037] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:35.619445 12644 raft_consensus.cc:2454] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35587c576db64103947f236cb6d071a1 in term 1.
I20260810 18:10:35.619445 12346 raft_consensus.cc:2454] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35587c576db64103947f236cb6d071a1 in term 1.
I20260810 18:10:35.619777 12423 leader_election.cc:302] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35587c576db64103947f236cb6d071a1, bf1f19591de147bd9dee03d0b2c292b7; no voters:
I20260810 18:10:35.620054 12726 raft_consensus.cc:2781] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:35.620262 12726 raft_consensus.cc:686] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [term 1 LEADER]: Becoming Leader. State: Replica: 35587c576db64103947f236cb6d071a1, State: Running, Role: LEADER
I20260810 18:10:35.620645 12726 consensus_queue.cc:237] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f19591de147bd9dee03d0b2c292b7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45801 } } peers { permanent_uuid: "35587c576db64103947f236cb6d071a1" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 32935 } } peers { permanent_uuid: "006c39d9d868403d9ee52c94a7f4670a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 41799 } }
I20260810 18:10:35.624496 12168 catalog_manager.cc:5495] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35587c576db64103947f236cb6d071a1 (127.21.182.66). New cstate: current_term: 1 leader_uuid: "35587c576db64103947f236cb6d071a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1f19591de147bd9dee03d0b2c292b7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35587c576db64103947f236cb6d071a1" member_type: VOTER last_known_addr { host: "127.21.182.66" port: 32935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "006c39d9d868403d9ee52c94a7f4670a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 41799 } health_report { overall_health: UNKNOWN } } }
I20260810 18:10:35.659390 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.655804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.21.182.66:32935)
0810 18:10:35.655996 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:35.656000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:35.656053 (+ 53us) server_negotiation.cc:407] Connection header received
0810 18:10:35.656213 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:35.656215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:35.656278 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:35.656369 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:10:35.656600 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.657084 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.657999 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:35.658202 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:35.658277 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:35.658578 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:10:35.659022 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:10:35.659116 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:35.659216 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 18:10:35.659264 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:10:35.666250 12346 raft_consensus.cc:1261] T 2f402d83589b454c851fd45bdc15439a P 006c39d9d868403d9ee52c94a7f4670a [term 1 FOLLOWER]: Refusing update from remote peer 35587c576db64103947f236cb6d071a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:10:35.666242 12644 raft_consensus.cc:1261] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7 [term 1 FOLLOWER]: Refusing update from remote peer 35587c576db64103947f236cb6d071a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:10:35.667086 12729 consensus_queue.cc:1035] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "006c39d9d868403d9ee52c94a7f4670a" member_type: VOTER last_known_addr { host: "127.21.182.65" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:10:35.667292 12726 consensus_queue.cc:1035] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf1f19591de147bd9dee03d0b2c292b7" member_type: VOTER last_known_addr { host: "127.21.182.67" port: 45801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:10:35.678926 12733 mvcc.cc:205] Tried to move back new op lower bound from 7317034744480387072 to 7317034744305160192. Current Snapshot: MvccSnapshot[applied={T|T < 7317034744480387072}]
I20260810 18:10:35.685314 12168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:10:35.685484 12168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:10:35.688159 12168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a954a9c8f434af898182abbfa81c4f9: Sending DeleteTablet for 3 replicas of tablet 2f402d83589b454c851fd45bdc15439a
I20260810 18:10:35.689122 12476 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f402d83589b454c851fd45bdc15439a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:35 UTC) from {username='slave'} at 127.0.0.1:44780
I20260810 18:10:35.689134 12622 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f402d83589b454c851fd45bdc15439a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:35 UTC) from {username='slave'} at 127.0.0.1:56084
I20260810 18:10:35.689707 12744 tablet_replica.cc:330] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1: stopping tablet replica
I20260810 18:10:35.689708 12319 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f402d83589b454c851fd45bdc15439a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:10:35 UTC) from {username='slave'} at 127.0.0.1:56334
I20260810 18:10:35.690006 12745 tablet_replica.cc:330] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7: stopping tablet replica
I20260810 18:10:35.690253 12744 raft_consensus.cc:2229] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:10:35.690330 12745 raft_consensus.cc:2229] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:10:35.690618 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 12246
I20260810 18:10:35.690706 12745 raft_consensus.cc:2258] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:35.692611 12744 raft_consensus.cc:2258] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:10:35.694828 12744 ts_tablet_manager.cc:1905] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:10:35.695518 12745 ts_tablet_manager.cc:1905] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:10:35.698073 12744 ts_tablet_manager.cc:1918] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:10:35.698220 12744 log.cc:1196] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-1/wal/wals/2f402d83589b454c851fd45bdc15439a
I20260810 18:10:35.698441 12745 ts_tablet_manager.cc:1918] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:10:35.698534 12745 log.cc:1196] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7: Deleting WAL directory at /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786385164390354-22233-0/minicluster-data/ts-2/wal/wals/2f402d83589b454c851fd45bdc15439a
I20260810 18:10:35.698606 12744 ts_tablet_manager.cc:1939] T 2f402d83589b454c851fd45bdc15439a P 35587c576db64103947f236cb6d071a1: Deleting consensus metadata
I20260810 18:10:35.698874 12745 ts_tablet_manager.cc:1939] T 2f402d83589b454c851fd45bdc15439a P bf1f19591de147bd9dee03d0b2c292b7: Deleting consensus metadata
I20260810 18:10:35.700316 12151 catalog_manager.cc:4841] TS 35587c576db64103947f236cb6d071a1 (127.21.182.66:32935): tablet 2f402d83589b454c851fd45bdc15439a (table test-table [id=0a31a376a4994ef4b4ad169de9eaadc8]) successfully deleted
I20260810 18:10:35.700565 12151 catalog_manager.cc:4841] TS bf1f19591de147bd9dee03d0b2c292b7 (127.21.182.67:45801): tablet 2f402d83589b454c851fd45bdc15439a (table test-table [id=0a31a376a4994ef4b4ad169de9eaadc8]) successfully deleted
W20260810 18:10:35.705703 12148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.182.65:41799 (error 108)
I20260810 18:10:35.705812 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 12405
I20260810 18:10:35.708583 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:35.708083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.65:41799 (local address 127.0.0.1:56340)
0810 18:10:35.708434 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 18:10:35.708498 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.65:41799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259}
W20260810 18:10:35.708966 12148 catalog_manager.cc:4584] TS 006c39d9d868403d9ee52c94a7f4670a (127.21.182.65:41799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f402d83589b454c851fd45bdc15439a: Network error: Client connection negotiation failed: client connection to 127.21.182.65:41799: connect: Connection refused (error 111)
I20260810 18:10:35.714351 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 12551
I20260810 18:10:35.724086 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 12132
2026-08-10T18:10:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1204 ms)
[----------] 6 tests from EncryptionPolicyTest (11611 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:35 dist-test-slave-27sr krb5kdc[12750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:35 dist-test-slave-27sr krb5kdc[12750](info): set up 2 sockets
Aug 10 18:10:35 dist-test-slave-27sr krb5kdc[12750](info): commencing operation
krb5kdc: starting...
W20260810 18:10:36.862350 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:36 dist-test-slave-27sr krb5kdc[12750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:10:37.004925 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41757
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:37.124688 12766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:37.125010 12766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:37.125054 12766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:37.129089 12766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:37.129156 12766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:37.129176 12766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:37.129194 12766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:37.129209 12766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:37.134724 12766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:37.135795 12766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:37.136914 12766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:37.146430 12772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:37.146430 12775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:37.146510 12766 server_base.cc:1034] running on GCE node
W20260810 18:10:37.146454 12771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:37.147352 12766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:37.148530 12766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:37.149820 12766 hybrid_clock.cc:648] HybridClock initialized: now 1786385437149779 us; error 53 us; skew 500 ppm
Aug 10 18:10:37 dist-test-slave-27sr krb5kdc[12750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:37.158349 12766 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:10:37.159348 12766 webserver.cc:460] Webserver started at http://127.0.0.1:42637/ using document root <none> and password file <none>
I20260810 18:10:37.160022 12766 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:37.160120 12766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:37.160413 12766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:37.162640 12766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "71173e6a63af4dd085f11a37f4dc9c0e"
format_stamp: "Formatted at 2026-08-10 18:10:37 on dist-test-slave-27sr"
server_key: "07db5af5a0ea00bdc45f8871b9636a44"
server_key_iv: "e9534f39806d8049c2aefb2d278d1108"
server_key_version: "encryptionkey@0"
I20260810 18:10:37.163231 12766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "71173e6a63af4dd085f11a37f4dc9c0e"
format_stamp: "Formatted at 2026-08-10 18:10:37 on dist-test-slave-27sr"
server_key: "07db5af5a0ea00bdc45f8871b9636a44"
server_key_iv: "e9534f39806d8049c2aefb2d278d1108"
server_key_version: "encryptionkey@0"
I20260810 18:10:37.166999 12766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:10:37.169560 12783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:37.170804 12766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:10:37.170948 12766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "71173e6a63af4dd085f11a37f4dc9c0e"
format_stamp: "Formatted at 2026-08-10 18:10:37 on dist-test-slave-27sr"
server_key: "07db5af5a0ea00bdc45f8871b9636a44"
server_key_iv: "e9534f39806d8049c2aefb2d278d1108"
server_key_version: "encryptionkey@0"
I20260810 18:10:37.171060 12766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:37.186352 12766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:37.189975 12766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:37.190212 12766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:37.197901 12766 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41757
I20260810 18:10:37.197932 12845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41757 every 8 connection(s)
I20260810 18:10:37.198977 12766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:37.202375 12846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:37.203023 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 12766
I20260810 18:10:37.203156 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:37.203532 22233 external_mini_cluster.cc:1428] Setting key 2df170df8ac02a97ee75a25b9349406e
I20260810 18:10:37.210748 12846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e: Bootstrap starting.
I20260810 18:10:37.214126 12846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:37.215011 12846 log.cc:824] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:10:37 dist-test-slave-27sr krb5kdc[12750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 18:10:37.217927 12846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e: No bootstrap required, opened a new log
I20260810 18:10:37.222486 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:37.205768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36931 (local address 127.0.0.1:41757)
0810 18:10:37.206333 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:37.206354 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:37.206405 (+ 51us) server_negotiation.cc:407] Connection header received
0810 18:10:37.207399 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:37.207430 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:37.207998 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:37.208469 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:37.210251 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:37.211698 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:37.212687 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:37.213215 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:37.213397 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:37.216968 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:37.216976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:37.216996 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:37.217034 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:37.219021 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:37.219702 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:37.219707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:37.219712 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:37.219873 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:37.220328 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:37.220331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:37.220332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:37.220747 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:37.220953 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:37.221339 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 18:10:37.221618 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":142,"threads_started":1}
I20260810 18:10:37.223052 12846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71173e6a63af4dd085f11a37f4dc9c0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41757 } }
I20260810 18:10:37.223351 12846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:37.223438 12846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71173e6a63af4dd085f11a37f4dc9c0e, State: Initialized, Role: FOLLOWER
I20260810 18:10:37.224006 12846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71173e6a63af4dd085f11a37f4dc9c0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41757 } }
I20260810 18:10:37.224167 12846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:37.224222 12846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:37.224313 12846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:37.225595 12846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71173e6a63af4dd085f11a37f4dc9c0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41757 } }
I20260810 18:10:37.226004 12846 leader_election.cc:302] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71173e6a63af4dd085f11a37f4dc9c0e; no voters:
I20260810 18:10:37.226181 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:37.226436 12846 leader_election.cc:288] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:37.226562 12852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:37.226783 12852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [term 1 LEADER]: Becoming Leader. State: Replica: 71173e6a63af4dd085f11a37f4dc9c0e, State: Running, Role: LEADER
I20260810 18:10:37.227162 12852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71173e6a63af4dd085f11a37f4dc9c0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41757 } }
I20260810 18:10:37.227933 12846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:37.228626 12853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71173e6a63af4dd085f11a37f4dc9c0e. Latest consensus state: current_term: 1 leader_uuid: "71173e6a63af4dd085f11a37f4dc9c0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71173e6a63af4dd085f11a37f4dc9c0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41757 } } }
I20260810 18:10:37.228770 12853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:37.229068 12855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71173e6a63af4dd085f11a37f4dc9c0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71173e6a63af4dd085f11a37f4dc9c0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41757 } } }
I20260810 18:10:37.229180 12855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:37.229254 12867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:37.232896 12867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:37.240481 12867 catalog_manager.cc:1347] Generated new cluster ID: d8afd05345f348cd8ef9e267095de7a1
I20260810 18:10:37.240561 12867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:37.241195 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:37.228734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41199 (local address 127.0.0.1:41757)
0810 18:10:37.228903 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:37.228911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:37.231171 (+ 2260us) server_negotiation.cc:407] Connection header received
0810 18:10:37.231533 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:37.231537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:37.231591 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:37.231688 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:37.232622 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:37.233366 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:37.234500 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:37.234859 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:37.234980 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:37.236873 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:37.236877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:37.236880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:37.236910 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:37.238927 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:37.239566 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:37.239570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:37.239572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:37.239670 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:37.240146 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:37.240149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:37.240151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:37.240395 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:37.240553 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:37.240927 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 18:10:37.241041 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 18:10:37.250787 12867 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:37.252192 12867 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:37.257342 12867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71173e6a63af4dd085f11a37f4dc9c0e: Generated new TSK 0
I20260810 18:10:37.258132 12867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:37.265014 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 12766
W20260810 18:10:37.274740 12748 connection.cc:535] client connection to 127.0.0.1:41757 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41757): Connection reset by peer (error 104)
2026-08-10T18:10:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1539 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:37 dist-test-slave-27sr krb5kdc[12884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:37 dist-test-slave-27sr krb5kdc[12884](info): set up 2 sockets
Aug 10 18:10:37 dist-test-slave-27sr krb5kdc[12884](info): commencing operation
krb5kdc: starting...
W20260810 18:10:38.397163 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:38 dist-test-slave-27sr krb5kdc[12884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:10:38.543880 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:38.667161 12897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:38.667538 12897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:38.667622 12897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:38.671945 12897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:38.672063 12897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:38.672102 12897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:38.672142 12897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:38.672178 12897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:38.677875 12897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33501
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:38.679162 12897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:38.680349 12897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:38.690378 12905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:38.690451 12903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:38.690429 12902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:38.690852 12897 server_base.cc:1034] running on GCE node
I20260810 18:10:38.691509 12897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:38.692951 12897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:38.694187 12897 hybrid_clock.cc:648] HybridClock initialized: now 1786385438694149 us; error 60 us; skew 500 ppm
Aug 10 18:10:38 dist-test-slave-27sr krb5kdc[12884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:38.697536 12897 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:10:38.698609 12897 webserver.cc:460] Webserver started at http://127.0.0.1:40819/ using document root <none> and password file <none>
I20260810 18:10:38.699251 12897 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:38.699350 12897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:38.699609 12897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:38.701900 12897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "3aadc0fc0a62449e9779480a684ce3f5"
format_stamp: "Formatted at 2026-08-10 18:10:38 on dist-test-slave-27sr"
server_key: "3a031afeb5f4dce5aa26aad7e21bf17b"
server_key_iv: "c78bb3dba61682a4bcd9c2048d9f7a2a"
server_key_version: "encryptionkey@0"
I20260810 18:10:38.702540 12897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "3aadc0fc0a62449e9779480a684ce3f5"
format_stamp: "Formatted at 2026-08-10 18:10:38 on dist-test-slave-27sr"
server_key: "3a031afeb5f4dce5aa26aad7e21bf17b"
server_key_iv: "c78bb3dba61682a4bcd9c2048d9f7a2a"
server_key_version: "encryptionkey@0"
I20260810 18:10:38.706740 12897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 18:10:38.709594 12914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:38.711050 12897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:10:38.711210 12897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "3aadc0fc0a62449e9779480a684ce3f5"
format_stamp: "Formatted at 2026-08-10 18:10:38 on dist-test-slave-27sr"
server_key: "3a031afeb5f4dce5aa26aad7e21bf17b"
server_key_iv: "c78bb3dba61682a4bcd9c2048d9f7a2a"
server_key_version: "encryptionkey@0"
I20260810 18:10:38.711342 12897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:38.765030 12897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:38.768656 12897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:38.768911 12897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:38.776798 12897 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33501
I20260810 18:10:38.776829 12976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33501 every 8 connection(s)
I20260810 18:10:38.777908 12897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:38.781181 12977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:38.783684 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 12897
I20260810 18:10:38.783854 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:38.784231 22233 external_mini_cluster.cc:1428] Setting key 102930d49fdef6cf800c80fdc831db51
I20260810 18:10:38.788698 12977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5: Bootstrap starting.
I20260810 18:10:38.791806 12977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:38.792865 12977 log.cc:824] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:38.795672 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:38.787009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48357 (local address 127.0.0.1:33501)
0810 18:10:38.787466 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:38.787485 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:38.787539 (+ 54us) server_negotiation.cc:407] Connection header received
0810 18:10:38.788270 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:38.788292 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:38.788679 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:38.789060 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:38.789430 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:38.790417 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:38.791657 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:38.792322 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:38.792452 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:38.793633 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:38.793640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:38.793660 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:38.794216 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:38.794285 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:38.794311 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:38.794428 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:38.794647 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 18:10:38.794824 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":148,"threads_started":1}
I20260810 18:10:38.795703 12977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5: No bootstrap required, opened a new log
I20260810 18:10:38.799293 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:38.800275 12977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aadc0fc0a62449e9779480a684ce3f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33501 } }
I20260810 18:10:38.800565 12977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:38.800638 12977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aadc0fc0a62449e9779480a684ce3f5, State: Initialized, Role: FOLLOWER
I20260810 18:10:38.801193 12977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aadc0fc0a62449e9779480a684ce3f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33501 } }
I20260810 18:10:38.801360 12977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:38.801453 12977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:38.801566 12977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:38.803020 12977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aadc0fc0a62449e9779480a684ce3f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33501 } }
I20260810 18:10:38.803426 12977 leader_election.cc:302] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aadc0fc0a62449e9779480a684ce3f5; no voters:
I20260810 18:10:38.803817 12977 leader_election.cc:288] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:38.804010 12988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:38.804236 12988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [term 1 LEADER]: Becoming Leader. State: Replica: 3aadc0fc0a62449e9779480a684ce3f5, State: Running, Role: LEADER
I20260810 18:10:38.804596 12988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aadc0fc0a62449e9779480a684ce3f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33501 } }
I20260810 18:10:38.805272 12977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:38.806079 12990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aadc0fc0a62449e9779480a684ce3f5. Latest consensus state: current_term: 1 leader_uuid: "3aadc0fc0a62449e9779480a684ce3f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aadc0fc0a62449e9779480a684ce3f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33501 } } }
I20260810 18:10:38.806236 12990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:38.806650 12989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aadc0fc0a62449e9779480a684ce3f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aadc0fc0a62449e9779480a684ce3f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33501 } } }
I20260810 18:10:38.806820 12989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:38.806869 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:38.801733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33279 (local address 127.0.0.1:33501)
0810 18:10:38.801887 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:38.801893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:38.802082 (+ 189us) server_negotiation.cc:407] Connection header received
0810 18:10:38.802177 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:38.802180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:38.802242 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:38.802334 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:38.802573 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:38.803373 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:38.804627 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:38.804953 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:38.805032 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:38.806266 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:38.806271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:38.806273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:38.806289 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:38.806321 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:38.806337 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:38.806435 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:38.806664 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 18:10:38.806703 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 18:10:38.810731 13006 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:10:38.810829 13006 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:10:38.810894 12996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:38.811794 12996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:38.818267 12996 catalog_manager.cc:1347] Generated new cluster ID: 45ddec0273234e8abadeff9ad23e0a14
I20260810 18:10:38.818365 12996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:38.825331 12996 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:38.826366 12996 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:38.838833 12996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aadc0fc0a62449e9779480a684ce3f5: Generated new TSK 0
I20260810 18:10:38.839691 12996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:38.847218 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 12897
W20260810 18:10:38.858348 12882 connection.cc:535] client connection to 127.0.0.1:33501 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33501): Connection reset by peer (error 104)
2026-08-10T18:10:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1583 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:38 dist-test-slave-27sr krb5kdc[13012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:38 dist-test-slave-27sr krb5kdc[13012](info): set up 2 sockets
Aug 10 18:10:38 dist-test-slave-27sr krb5kdc[13012](info): commencing operation
krb5kdc: starting...
W20260810 18:10:39.978597 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:40 dist-test-slave-27sr krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:10:40.123607 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:40.241531 13027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:40.241833 13027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:40.241875 13027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:40.246042 13027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:40.246114 13027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:40.246133 13027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:40.246151 13027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:40.246166 13027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:40.251718 13027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:40.253068 13027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:40.254240 13027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:40.263535 13032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:40.263514 13035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:40.263514 13037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:40.264112 13027 server_base.cc:1034] running on GCE node
I20260810 18:10:40.264806 13027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:40.266137 13027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:40.267336 13027 hybrid_clock.cc:648] HybridClock initialized: now 1786385440267314 us; error 32 us; skew 500 ppm
Aug 10 18:10:40 dist-test-slave-27sr krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:40.270646 13027 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:10:40.271788 13027 webserver.cc:460] Webserver started at http://127.0.0.1:46727/ using document root <none> and password file <none>
I20260810 18:10:40.272475 13027 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:40.272543 13027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:40.272790 13027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:40.275013 13027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "86c75388fa4b4ab7a2daff61a1cf2d78"
format_stamp: "Formatted at 2026-08-10 18:10:40 on dist-test-slave-27sr"
server_key: "60869774e209a8a25f1892dc9c0f6f00"
server_key_iv: "376da290c65df16a667479ece775b0c9"
server_key_version: "encryptionkey@0"
I20260810 18:10:40.275609 13027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "86c75388fa4b4ab7a2daff61a1cf2d78"
format_stamp: "Formatted at 2026-08-10 18:10:40 on dist-test-slave-27sr"
server_key: "60869774e209a8a25f1892dc9c0f6f00"
server_key_iv: "376da290c65df16a667479ece775b0c9"
server_key_version: "encryptionkey@0"
I20260810 18:10:40.279639 13027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:10:40.282366 13044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:40.283718 13027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:10:40.283885 13027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "86c75388fa4b4ab7a2daff61a1cf2d78"
format_stamp: "Formatted at 2026-08-10 18:10:40 on dist-test-slave-27sr"
server_key: "60869774e209a8a25f1892dc9c0f6f00"
server_key_iv: "376da290c65df16a667479ece775b0c9"
server_key_version: "encryptionkey@0"
I20260810 18:10:40.284062 13027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:40.288973 13027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:40.292852 13027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:40.293061 13027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:40.300823 13027 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36033
I20260810 18:10:40.300835 13102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36033 every 8 connection(s)
I20260810 18:10:40.301823 13027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:40.304859 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:40.310484 13103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78: Bootstrap starting.
I20260810 18:10:40.311453 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 13027
I20260810 18:10:40.311612 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:40.311997 22233 external_mini_cluster.cc:1428] Setting key 4aacbd5ec82382887532b8f6b625452a
I20260810 18:10:40.313577 13103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:40.314337 13103 log.cc:824] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:40.316802 13103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78: No bootstrap required, opened a new log
I20260810 18:10:40.317443 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:40.313172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34343 (local address 127.0.0.1:36033)
0810 18:10:40.313673 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:40.313696 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:40.313755 (+ 59us) server_negotiation.cc:407] Connection header received
0810 18:10:40.314390 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:40.314436 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:40.314803 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:40.315186 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:40.316101 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:40.316106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:40.316118 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:40.316129 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:40.316185 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:40.316204 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:40.316300 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:40.316436 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 18:10:40.316608 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":139,"threads_started":1}
I20260810 18:10:40.320370 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:40.320566 13103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c75388fa4b4ab7a2daff61a1cf2d78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36033 } }
I20260810 18:10:40.320789 13103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:40.320869 13103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86c75388fa4b4ab7a2daff61a1cf2d78, State: Initialized, Role: FOLLOWER
I20260810 18:10:40.321324 13103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c75388fa4b4ab7a2daff61a1cf2d78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36033 } }
I20260810 18:10:40.321450 13103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:40.321534 13103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:40.321648 13103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:40.322695 13103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c75388fa4b4ab7a2daff61a1cf2d78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36033 } }
I20260810 18:10:40.323058 13103 leader_election.cc:302] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86c75388fa4b4ab7a2daff61a1cf2d78; no voters:
I20260810 18:10:40.323362 13103 leader_election.cc:288] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:40.323479 13115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:40.324043 13115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [term 1 LEADER]: Becoming Leader. State: Replica: 86c75388fa4b4ab7a2daff61a1cf2d78, State: Running, Role: LEADER
I20260810 18:10:40.324375 13115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c75388fa4b4ab7a2daff61a1cf2d78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36033 } }
I20260810 18:10:40.324750 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:40.322708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46331 (local address 127.0.0.1:36033)
0810 18:10:40.322839 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:40.322844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:40.323078 (+ 234us) server_negotiation.cc:407] Connection header received
0810 18:10:40.323297 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:40.323300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:40.323335 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:40.323422 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:40.324224 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:40.324228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:40.324230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:40.324243 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:40.324271 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:40.324285 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:40.324344 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:40.324586 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 18:10:40.324629 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 18:10:40.324847 13103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:40.325706 13117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86c75388fa4b4ab7a2daff61a1cf2d78. Latest consensus state: current_term: 1 leader_uuid: "86c75388fa4b4ab7a2daff61a1cf2d78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c75388fa4b4ab7a2daff61a1cf2d78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36033 } } }
I20260810 18:10:40.325850 13117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:40.326198 13116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86c75388fa4b4ab7a2daff61a1cf2d78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c75388fa4b4ab7a2daff61a1cf2d78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36033 } } }
I20260810 18:10:40.326314 13116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78 [sys.catalog]: This master's current role is: LEADER
W20260810 18:10:40.329532 13133 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:10:40.329625 13133 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:10:40.329704 13126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:40.330423 13126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:40.336315 13126 catalog_manager.cc:1347] Generated new cluster ID: 6dd396b7dca44a28b3a42ed2995b1997
I20260810 18:10:40.336388 13126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:40.342834 13126 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:40.343178 13126 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:40.353492 13126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86c75388fa4b4ab7a2daff61a1cf2d78: Generated new TSK 0
I20260810 18:10:40.354441 13126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:40.358423 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 13027
2026-08-10T18:10:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1507 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:40 dist-test-slave-27sr krb5kdc[13141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:40 dist-test-slave-27sr krb5kdc[13141](info): set up 2 sockets
Aug 10 18:10:40 dist-test-slave-27sr krb5kdc[13141](info): commencing operation
krb5kdc: starting...
W20260810 18:10:41.519151 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:41 dist-test-slave-27sr krb5kdc[13141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:10:41.676205 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:41.801347 13156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:41.801625 13156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:41.801668 13156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:41.805970 13156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:41.806036 13156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:41.806056 13156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:41.806073 13156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:41.806088 13156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:41.811456 13156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:41.812572 13156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:41.813589 13156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:41.822911 13164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:41.822914 13165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:41.822975 13168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:41.823251 13156 server_base.cc:1034] running on GCE node
I20260810 18:10:41.824059 13156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:41.825254 13156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:41.826436 13156 hybrid_clock.cc:648] HybridClock initialized: now 1786385441826423 us; error 28 us; skew 500 ppm
Aug 10 18:10:41 dist-test-slave-27sr krb5kdc[13141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:41.829704 13156 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:10:41.830789 13156 webserver.cc:460] Webserver started at http://127.0.0.1:43315/ using document root <none> and password file <none>
I20260810 18:10:41.831385 13156 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:41.831453 13156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:41.831676 13156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:41.833878 13156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "7d79f501e09f4ccb80891ae7500b66b6"
format_stamp: "Formatted at 2026-08-10 18:10:41 on dist-test-slave-27sr"
server_key: "b734bac1ded6a905ca9acab343dfbca0"
server_key_iv: "2c771a0116b0759c5a0b83227edc8db5"
server_key_version: "encryptionkey@0"
I20260810 18:10:41.834537 13156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "7d79f501e09f4ccb80891ae7500b66b6"
format_stamp: "Formatted at 2026-08-10 18:10:41 on dist-test-slave-27sr"
server_key: "b734bac1ded6a905ca9acab343dfbca0"
server_key_iv: "2c771a0116b0759c5a0b83227edc8db5"
server_key_version: "encryptionkey@0"
I20260810 18:10:41.838945 13156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:10:41.841737 13178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:41.843053 13156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:10:41.843201 13156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "7d79f501e09f4ccb80891ae7500b66b6"
format_stamp: "Formatted at 2026-08-10 18:10:41 on dist-test-slave-27sr"
server_key: "b734bac1ded6a905ca9acab343dfbca0"
server_key_iv: "2c771a0116b0759c5a0b83227edc8db5"
server_key_version: "encryptionkey@0"
I20260810 18:10:41.843313 13156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:41.859412 13156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:41.862984 13156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:41.863219 13156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:41.871528 13156 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46259
I20260810 18:10:41.871555 13235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46259 every 8 connection(s)
I20260810 18:10:41.872592 13156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:41.874953 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 13156
I20260810 18:10:41.875121 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:41.875429 22233 external_mini_cluster.cc:1428] Setting key 9d1e90ebf4fc832fe0b0e09969f5968a
I20260810 18:10:41.875553 13236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:41.882279 13236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6: Bootstrap starting.
I20260810 18:10:41.884963 13236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6: Neither blocks nor log segments found. Creating new log.
Aug 10 18:10:41 dist-test-slave-27sr krb5kdc[13141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 18:10:41.885763 13236 log.cc:824] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:41.888170 13236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6: No bootstrap required, opened a new log
I20260810 18:10:41.890798 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:41.877051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40109 (local address 127.0.0.1:46259)
0810 18:10:41.877548 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:41.877564 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:41.877601 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:10:41.878399 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:41.878418 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:41.878775 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:41.879146 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:41.880119 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:41.881159 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:41.882083 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:41.882619 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:41.882800 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:41.885796 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:41.885802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:41.885817 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:41.885873 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:41.887769 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:41.888289 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:41.888295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:41.888308 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:41.888425 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:41.888744 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:41.888748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:41.888749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:41.889179 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:41.889397 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:41.889714 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 18:10:41.890038 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":162,"threads_started":1}
I20260810 18:10:41.892302 13236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d79f501e09f4ccb80891ae7500b66b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46259 } }
I20260810 18:10:41.892556 13236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:41.892638 13236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d79f501e09f4ccb80891ae7500b66b6, State: Initialized, Role: FOLLOWER
I20260810 18:10:41.893136 13236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d79f501e09f4ccb80891ae7500b66b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46259 } }
I20260810 18:10:41.893306 13236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:41.893394 13236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:41.893507 13236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:41.894192 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:41.894724 13236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d79f501e09f4ccb80891ae7500b66b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46259 } }
I20260810 18:10:41.895074 13236 leader_election.cc:302] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d79f501e09f4ccb80891ae7500b66b6; no voters:
I20260810 18:10:41.895428 13236 leader_election.cc:288] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:41.895529 13244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:41.895746 13244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [term 1 LEADER]: Becoming Leader. State: Replica: 7d79f501e09f4ccb80891ae7500b66b6, State: Running, Role: LEADER
I20260810 18:10:41.896071 13244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d79f501e09f4ccb80891ae7500b66b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46259 } }
I20260810 18:10:41.896814 13236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:41.897305 13244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d79f501e09f4ccb80891ae7500b66b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d79f501e09f4ccb80891ae7500b66b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46259 } } }
I20260810 18:10:41.897421 13244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:41.897645 13244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d79f501e09f4ccb80891ae7500b66b6. Latest consensus state: current_term: 1 leader_uuid: "7d79f501e09f4ccb80891ae7500b66b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d79f501e09f4ccb80891ae7500b66b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46259 } } }
I20260810 18:10:41.897763 13244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:41.897998 13256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:41.901520 13256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:41.906371 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:41.896414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33581 (local address 127.0.0.1:46259)
0810 18:10:41.896534 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:41.896538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:41.896803 (+ 265us) server_negotiation.cc:407] Connection header received
0810 18:10:41.896972 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:41.896975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:41.897020 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:41.897116 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:41.897983 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:41.898845 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:41.900017 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:41.900346 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:41.900439 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:41.902601 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:41.902605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:41.902609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:41.902636 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:41.904467 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:41.904983 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:41.904987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:41.904989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:41.905083 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:41.905402 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:41.905406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:41.905408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:41.905642 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:41.905785 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:41.906093 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 18:10:41.906216 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:10:41.908085 13256 catalog_manager.cc:1347] Generated new cluster ID: 4f74fa167fe5421aaae456f422a883a6
I20260810 18:10:41.908154 13256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:41.917309 13256 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:41.918337 13256 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:41.931490 13256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d79f501e09f4ccb80891ae7500b66b6: Generated new TSK 0
I20260810 18:10:41.932488 13256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:41.945845 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 13156
2026-08-10T18:10:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1592 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:42 dist-test-slave-27sr krb5kdc[13274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:42 dist-test-slave-27sr krb5kdc[13274](info): set up 2 sockets
Aug 10 18:10:42 dist-test-slave-27sr krb5kdc[13274](info): commencing operation
krb5kdc: starting...
W20260810 18:10:43.065680 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:43 dist-test-slave-27sr krb5kdc[13274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:10:43.203693 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37301
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:43.326472 13292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:43.326828 13292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:43.326942 13292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:43.331161 13292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:43.331267 13292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:43.331305 13292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:43.331341 13292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:43.331380 13292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:43.337072 13292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:43.338346 13292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:43.339516 13292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:43.349337 13298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:43.349367 13297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:43.349337 13300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:43.349772 13292 server_base.cc:1034] running on GCE node
I20260810 18:10:43.350495 13292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:43.351850 13292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:43.353091 13292 hybrid_clock.cc:648] HybridClock initialized: now 1786385443353075 us; error 34 us; skew 500 ppm
Aug 10 18:10:43 dist-test-slave-27sr krb5kdc[13274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:43.361927 13292 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:10:43.363070 13292 webserver.cc:460] Webserver started at http://127.0.0.1:36773/ using document root <none> and password file <none>
I20260810 18:10:43.363711 13292 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:43.363781 13292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:43.364058 13292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:43.366518 13292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "7951587a44a1443e9a06972148dc88a3"
format_stamp: "Formatted at 2026-08-10 18:10:43 on dist-test-slave-27sr"
server_key: "bee208086055a5baef7babb4b30bcfe1"
server_key_iv: "f072224f710861461b7c499628132493"
server_key_version: "encryptionkey@0"
I20260810 18:10:43.367128 13292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "7951587a44a1443e9a06972148dc88a3"
format_stamp: "Formatted at 2026-08-10 18:10:43 on dist-test-slave-27sr"
server_key: "bee208086055a5baef7babb4b30bcfe1"
server_key_iv: "f072224f710861461b7c499628132493"
server_key_version: "encryptionkey@0"
I20260810 18:10:43.371536 13292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:10:43.374624 13311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:43.376196 13292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:43.376324 13292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "7951587a44a1443e9a06972148dc88a3"
format_stamp: "Formatted at 2026-08-10 18:10:43 on dist-test-slave-27sr"
server_key: "bee208086055a5baef7babb4b30bcfe1"
server_key_iv: "f072224f710861461b7c499628132493"
server_key_version: "encryptionkey@0"
I20260810 18:10:43.376458 13292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:43.394644 13292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:43.397758 13292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:43.397931 13292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:43.406100 13292 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37301
I20260810 18:10:43.406123 13374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37301 every 8 connection(s)
I20260810 18:10:43.407148 13292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:43.410087 13376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:43.412221 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 13292
I20260810 18:10:43.412384 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:43.412670 22233 external_mini_cluster.cc:1428] Setting key 94c822224a7f8f90c551819e9921e5cb
I20260810 18:10:43.417305 13376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3: Bootstrap starting.
I20260810 18:10:43.419641 13376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:43.420486 13376 log.cc:824] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:43.422259 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:43.414199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37671 (local address 127.0.0.1:37301)
0810 18:10:43.414727 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:43.414752 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:43.414805 (+ 53us) server_negotiation.cc:407] Connection header received
0810 18:10:43.415583 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:43.415605 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:43.416034 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:43.416389 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:43.416805 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:43.417861 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:43.418818 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:43.419304 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:43.419507 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:43.420469 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:43.420475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:43.420499 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:43.420962 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:43.421020 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:43.421044 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:43.421162 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:43.421292 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 18:10:43.421530 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":147,"threads_started":1}
I20260810 18:10:43.422727 13376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3: No bootstrap required, opened a new log
I20260810 18:10:43.425503 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:43.427016 13376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7951587a44a1443e9a06972148dc88a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37301 } }
I20260810 18:10:43.427327 13376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:43.427403 13376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7951587a44a1443e9a06972148dc88a3, State: Initialized, Role: FOLLOWER
I20260810 18:10:43.428124 13376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7951587a44a1443e9a06972148dc88a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37301 } }
I20260810 18:10:43.428306 13376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:43.428409 13376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:43.428567 13376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:43.429808 13376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7951587a44a1443e9a06972148dc88a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37301 } }
I20260810 18:10:43.430191 13376 leader_election.cc:302] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7951587a44a1443e9a06972148dc88a3; no voters:
I20260810 18:10:43.430564 13376 leader_election.cc:288] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:43.430689 13388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:43.430933 13388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [term 1 LEADER]: Becoming Leader. State: Replica: 7951587a44a1443e9a06972148dc88a3, State: Running, Role: LEADER
I20260810 18:10:43.431285 13388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7951587a44a1443e9a06972148dc88a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37301 } }
I20260810 18:10:43.432129 13376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:43.432344 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:43.427589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51923 (local address 127.0.0.1:37301)
0810 18:10:43.427743 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:43.427748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:43.427986 (+ 238us) server_negotiation.cc:407] Connection header received
0810 18:10:43.428158 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:43.428161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:43.428216 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:43.428303 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:43.428618 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:43.429372 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:43.430264 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:43.430666 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:43.430806 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:43.431802 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:43.431806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:43.431809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:43.431822 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:43.431852 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:43.431866 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:43.431964 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:43.432089 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 18:10:43.432196 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:10:43.432783 13388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7951587a44a1443e9a06972148dc88a3. Latest consensus state: current_term: 1 leader_uuid: "7951587a44a1443e9a06972148dc88a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7951587a44a1443e9a06972148dc88a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37301 } } }
I20260810 18:10:43.432912 13388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:43.433312 13395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:43.433708 13390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7951587a44a1443e9a06972148dc88a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7951587a44a1443e9a06972148dc88a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37301 } } }
I20260810 18:10:43.433828 13390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:43.436672 13395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:43.444973 13395 catalog_manager.cc:1347] Generated new cluster ID: 579331c4320b4e15bb4711a47b3a6653
I20260810 18:10:43.445070 13395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:43.461047 13395 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:43.463214 13395 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:43.475821 13395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7951587a44a1443e9a06972148dc88a3: Generated new TSK 0
I20260810 18:10:43.476715 13395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:43.488567 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 13292
2026-08-10T18:10:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1538 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:43 dist-test-slave-27sr krb5kdc[13411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:43 dist-test-slave-27sr krb5kdc[13411](info): set up 2 sockets
Aug 10 18:10:43 dist-test-slave-27sr krb5kdc[13411](info): commencing operation
krb5kdc: starting...
W20260810 18:10:44.609692 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:44 dist-test-slave-27sr krb5kdc[13411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:10:44.751153 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:44.872488 13424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:44.872848 13424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:44.872941 13424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:44.877161 13424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:44.877269 13424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:44.877313 13424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:44.877370 13424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:44.877410 13424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:44.882952 13424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:44.884182 13424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:44.885324 13424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:10:44.895040 13424 server_base.cc:1034] running on GCE node
W20260810 18:10:44.895023 13436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:44.894999 13434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:44.894999 13432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:44.896001 13424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:44.897212 13424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:44.898417 13424 hybrid_clock.cc:648] HybridClock initialized: now 1786385444898403 us; error 29 us; skew 500 ppm
Aug 10 18:10:44 dist-test-slave-27sr krb5kdc[13411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:44.901512 13424 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:10:44.902612 13424 webserver.cc:460] Webserver started at http://127.0.0.1:40965/ using document root <none> and password file <none>
I20260810 18:10:44.903250 13424 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:44.903369 13424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:44.903609 13424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:44.905877 13424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "cb706c8d1427403aa4ec6fbdccc1f945"
format_stamp: "Formatted at 2026-08-10 18:10:44 on dist-test-slave-27sr"
server_key: "49ca4b5d15765a226c81ea7ac8535442"
server_key_iv: "e23446b011a23da812870fdb39959b37"
server_key_version: "encryptionkey@0"
I20260810 18:10:44.906481 13424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "cb706c8d1427403aa4ec6fbdccc1f945"
format_stamp: "Formatted at 2026-08-10 18:10:44 on dist-test-slave-27sr"
server_key: "49ca4b5d15765a226c81ea7ac8535442"
server_key_iv: "e23446b011a23da812870fdb39959b37"
server_key_version: "encryptionkey@0"
I20260810 18:10:44.910646 13424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 18:10:44.913359 13443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:44.914786 13424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:10:44.914961 13424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "cb706c8d1427403aa4ec6fbdccc1f945"
format_stamp: "Formatted at 2026-08-10 18:10:44 on dist-test-slave-27sr"
server_key: "49ca4b5d15765a226c81ea7ac8535442"
server_key_iv: "e23446b011a23da812870fdb39959b37"
server_key_version: "encryptionkey@0"
I20260810 18:10:44.915086 13424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:44.920279 13424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:44.923740 13424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:44.924031 13424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:44.932453 13424 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32889
I20260810 18:10:44.932477 13505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32889 every 8 connection(s)
I20260810 18:10:44.933816 13424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:44.937965 13506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:44.939213 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 13424
I20260810 18:10:44.939381 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:44.939704 22233 external_mini_cluster.cc:1428] Setting key 63e061773f5c700846abc050e2797e68
I20260810 18:10:44.946039 13506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945: Bootstrap starting.
I20260810 18:10:44.946847 13509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:44.941964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60767 (local address 127.0.0.1:32889)
0810 18:10:44.942588 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:44.942614 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:44.942669 (+ 55us) server_negotiation.cc:407] Connection header received
0810 18:10:44.943568 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:44.943590 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:44.944010 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:44.944424 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:44.945376 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:44.945388 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:44.945402 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:44.945415 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:44.945477 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:44.945500 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:44.945672 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:44.945789 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 18:10:44.945981 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":162,"threads_started":1}
I20260810 18:10:44.948881 13506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:44.949816 13506 log.cc:824] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:44.950371 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:44.952729 13506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945: No bootstrap required, opened a new log
I20260810 18:10:44.954659 13509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:44.952845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44961 (local address 127.0.0.1:32889)
0810 18:10:44.952968 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:44.952973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:44.953213 (+ 240us) server_negotiation.cc:407] Connection header received
0810 18:10:44.953297 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:44.953300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:44.953337 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:44.953440 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:44.954251 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:44.954254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:44.954257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:44.954269 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:44.954301 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:44.954314 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:44.954419 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:44.954482 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 18:10:44.954524 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 18:10:44.957346 13506 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb706c8d1427403aa4ec6fbdccc1f945" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } }
I20260810 18:10:44.957612 13506 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:44.957650 13506 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb706c8d1427403aa4ec6fbdccc1f945, State: Initialized, Role: FOLLOWER
I20260810 18:10:44.958109 13506 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb706c8d1427403aa4ec6fbdccc1f945" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } }
I20260810 18:10:44.958247 13506 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:44.958307 13506 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:44.958389 13506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:44.959471 13506 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb706c8d1427403aa4ec6fbdccc1f945" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } }
I20260810 18:10:44.959815 13506 leader_election.cc:302] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb706c8d1427403aa4ec6fbdccc1f945; no voters:
I20260810 18:10:44.960271 13506 leader_election.cc:288] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:44.960414 13518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:44.960672 13518 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [term 1 LEADER]: Becoming Leader. State: Replica: cb706c8d1427403aa4ec6fbdccc1f945, State: Running, Role: LEADER
I20260810 18:10:44.961076 13518 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb706c8d1427403aa4ec6fbdccc1f945" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } }
I20260810 18:10:44.961750 13506 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:44.963161 13520 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb706c8d1427403aa4ec6fbdccc1f945. Latest consensus state: current_term: 1 leader_uuid: "cb706c8d1427403aa4ec6fbdccc1f945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb706c8d1427403aa4ec6fbdccc1f945" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } } }
I20260810 18:10:44.963330 13520 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:44.963157 13519 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb706c8d1427403aa4ec6fbdccc1f945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb706c8d1427403aa4ec6fbdccc1f945" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32889 } } }
I20260810 18:10:44.963440 13519 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:44.963800 13528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:44.967356 13528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:44.973905 13528 catalog_manager.cc:1347] Generated new cluster ID: bcd832a1eabe4960a2d18db0153c0728
I20260810 18:10:44.974009 13528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:44.983294 13528 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:44.983678 13528 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:44.991927 13528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb706c8d1427403aa4ec6fbdccc1f945: Generated new TSK 0
I20260810 18:10:44.992977 13528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:45.006492 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 13424
2026-08-10T18:10:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1520 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:45 dist-test-slave-27sr krb5kdc[13543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:45 dist-test-slave-27sr krb5kdc[13543](info): set up 2 sockets
Aug 10 18:10:45 dist-test-slave-27sr krb5kdc[13543](info): commencing operation
krb5kdc: starting...
W20260810 18:10:46.191036 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:46 dist-test-slave-27sr krb5kdc[13543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:10:46.331820 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38389
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:46.454685 13557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:46.454984 13557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:46.455029 13557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:46.459278 13557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:46.459342 13557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:46.459362 13557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:46.459378 13557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:46.459415 13557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:46.465056 13557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:46.466233 13557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:46.467314 13557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:46.476725 13564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:46.476768 13557 server_base.cc:1034] running on GCE node
W20260810 18:10:46.476683 13565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:46.476691 13567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:46.477659 13557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:46.478927 13557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:46.480098 13557 hybrid_clock.cc:648] HybridClock initialized: now 1786385446480076 us; error 36 us; skew 500 ppm
Aug 10 18:10:46 dist-test-slave-27sr krb5kdc[13543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:46.483014 13557 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:10:46.484126 13557 webserver.cc:460] Webserver started at http://127.0.0.1:37727/ using document root <none> and password file <none>
I20260810 18:10:46.484735 13557 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:46.484833 13557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:46.485088 13557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:46.487228 13557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "c7398e50ce6c4c86b602eb3e117bc5f8"
format_stamp: "Formatted at 2026-08-10 18:10:46 on dist-test-slave-27sr"
server_key: "4f17c70875a2068fc7cb2803ef4571d3"
server_key_iv: "80e0758b77dbde1bb16c607b6b0681ff"
server_key_version: "encryptionkey@0"
I20260810 18:10:46.487833 13557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "c7398e50ce6c4c86b602eb3e117bc5f8"
format_stamp: "Formatted at 2026-08-10 18:10:46 on dist-test-slave-27sr"
server_key: "4f17c70875a2068fc7cb2803ef4571d3"
server_key_iv: "80e0758b77dbde1bb16c607b6b0681ff"
server_key_version: "encryptionkey@0"
I20260810 18:10:46.491845 13557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 18:10:46.494742 13575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:46.496227 13557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:46.496407 13557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "c7398e50ce6c4c86b602eb3e117bc5f8"
format_stamp: "Formatted at 2026-08-10 18:10:46 on dist-test-slave-27sr"
server_key: "4f17c70875a2068fc7cb2803ef4571d3"
server_key_iv: "80e0758b77dbde1bb16c607b6b0681ff"
server_key_version: "encryptionkey@0"
I20260810 18:10:46.496543 13557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:46.518040 13557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:46.521565 13557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:46.521807 13557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:46.530025 13557 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38389
I20260810 18:10:46.530036 13637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38389 every 8 connection(s)
I20260810 18:10:46.531111 13557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:46.534273 13638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:46.539954 13638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8: Bootstrap starting.
I20260810 18:10:46.540634 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 13557
I20260810 18:10:46.540795 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:46.541074 22233 external_mini_cluster.cc:1428] Setting key 653ded225f882ca5ede10229c56f5bf9
I20260810 18:10:46.542887 13638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:46.543658 13638 log.cc:824] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:46.546190 13638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8: No bootstrap required, opened a new log
Aug 10 18:10:46 dist-test-slave-27sr krb5kdc[13543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 18:10:46.550688 13638 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7398e50ce6c4c86b602eb3e117bc5f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38389 } }
I20260810 18:10:46.551038 13638 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:46.551126 13638 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7398e50ce6c4c86b602eb3e117bc5f8, State: Initialized, Role: FOLLOWER
I20260810 18:10:46.551672 13638 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7398e50ce6c4c86b602eb3e117bc5f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38389 } }
I20260810 18:10:46.551851 13638 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:46.551971 13638 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:46.552094 13638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:46.553344 13638 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7398e50ce6c4c86b602eb3e117bc5f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38389 } }
I20260810 18:10:46.553742 13638 leader_election.cc:302] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7398e50ce6c4c86b602eb3e117bc5f8; no voters:
I20260810 18:10:46.554159 13638 leader_election.cc:288] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:46.554282 13643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:46.554531 13643 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [term 1 LEADER]: Becoming Leader. State: Replica: c7398e50ce6c4c86b602eb3e117bc5f8, State: Running, Role: LEADER
I20260810 18:10:46.555038 13643 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7398e50ce6c4c86b602eb3e117bc5f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38389 } }
I20260810 18:10:46.555676 13638 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:46.557264 13645 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7398e50ce6c4c86b602eb3e117bc5f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7398e50ce6c4c86b602eb3e117bc5f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38389 } } }
I20260810 18:10:46.557427 13645 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:46.557788 13646 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7398e50ce6c4c86b602eb3e117bc5f8. Latest consensus state: current_term: 1 leader_uuid: "c7398e50ce6c4c86b602eb3e117bc5f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7398e50ce6c4c86b602eb3e117bc5f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38389 } } }
I20260810 18:10:46.557958 13646 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:46.558269 13655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:46.558290 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:46.542469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:52899 (local address 127.0.0.1:38389)
0810 18:10:46.542944 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:46.542965 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:46.543019 (+ 54us) server_negotiation.cc:407] Connection header received
0810 18:10:46.543700 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:46.543718 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:46.544203 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:46.544615 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:46.545335 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:46.546802 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:46.547691 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:46.548457 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:46.548627 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:46.551567 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:46.551572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:46.551589 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:46.551621 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:46.553750 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:46.554646 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:46.554653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:46.554659 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:46.554817 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:46.555276 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:46.555279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:46.555281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:46.555989 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:46.556262 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:46.556654 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 18:10:46.556956 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":152,"threads_started":1}
I20260810 18:10:46.562057 13655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:46.562348 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:46.569674 13655 catalog_manager.cc:1347] Generated new cluster ID: 4bdf90391bc24202b9c34af0bff561ca
I20260810 18:10:46.569826 13655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:46.576310 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:46.564808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:55665 (local address 127.0.0.1:38389)
0810 18:10:46.564926 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:46.564931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:46.566631 (+ 1700us) server_negotiation.cc:407] Connection header received
0810 18:10:46.566895 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:46.566898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:46.566962 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:46.567061 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:46.567851 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:46.568784 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:46.570044 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:46.570411 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:46.570526 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:46.572444 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:46.572448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:46.572451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:46.572481 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:46.574329 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:46.574910 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:46.574914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:46.574916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:46.575007 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:46.575357 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:46.575360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:46.575362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:46.575580 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:46.575733 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:46.576036 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 18:10:46.576146 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:10:46.599108 13655 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:46.600598 13655 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:46.608500 13655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7398e50ce6c4c86b602eb3e117bc5f8: Generated new TSK 0
I20260810 18:10:46.609504 13655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:46.614861 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 13557
W20260810 18:10:46.626430 13541 connection.cc:535] client connection to 127.0.0.1:38389 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38389): Connection reset by peer (error 104)
2026-08-10T18:10:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1609 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:46 dist-test-slave-27sr krb5kdc[13674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:46 dist-test-slave-27sr krb5kdc[13674](info): set up 2 sockets
Aug 10 18:10:46 dist-test-slave-27sr krb5kdc[13674](info): commencing operation
krb5kdc: starting...
W20260810 18:10:47.747411 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:47 dist-test-slave-27sr krb5kdc[13674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:10:47.894968 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:48.017565 13690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:48.017915 13690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:48.018002 13690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:48.022186 13690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:48.022302 13690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:48.022356 13690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:48.022418 13690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:48.022459 13690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:48.028026 13690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:48.029202 13690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:48.030270 13690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:48.039312 13696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:48.039320 13697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:48.039323 13699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:48.039556 13690 server_base.cc:1034] running on GCE node
I20260810 18:10:48.040259 13690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:48.041427 13690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:48.042596 13690 hybrid_clock.cc:648] HybridClock initialized: now 1786385448042575 us; error 33 us; skew 500 ppm
Aug 10 18:10:48 dist-test-slave-27sr krb5kdc[13674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:48.045625 13690 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:10:48.046636 13690 webserver.cc:460] Webserver started at http://127.0.0.1:44463/ using document root <none> and password file <none>
I20260810 18:10:48.047224 13690 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:48.047289 13690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:48.047549 13690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:48.049839 13690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "252c69b9027845659eb429852faa8609"
format_stamp: "Formatted at 2026-08-10 18:10:48 on dist-test-slave-27sr"
server_key: "5adebb47993236b91755051a5e4ae4b5"
server_key_iv: "94a20077fe596ed65f299f2e5514bfd9"
server_key_version: "encryptionkey@0"
I20260810 18:10:48.050455 13690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "252c69b9027845659eb429852faa8609"
format_stamp: "Formatted at 2026-08-10 18:10:48 on dist-test-slave-27sr"
server_key: "5adebb47993236b91755051a5e4ae4b5"
server_key_iv: "94a20077fe596ed65f299f2e5514bfd9"
server_key_version: "encryptionkey@0"
I20260810 18:10:48.054524 13690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:10:48.057241 13706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:48.058692 13690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:10:48.058881 13690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "252c69b9027845659eb429852faa8609"
format_stamp: "Formatted at 2026-08-10 18:10:48 on dist-test-slave-27sr"
server_key: "5adebb47993236b91755051a5e4ae4b5"
server_key_iv: "94a20077fe596ed65f299f2e5514bfd9"
server_key_version: "encryptionkey@0"
I20260810 18:10:48.059029 13690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:48.078244 13690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:48.082170 13690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:48.082417 13690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:48.090509 13690 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39729
I20260810 18:10:48.090539 13776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39729 every 8 connection(s)
I20260810 18:10:48.091600 13690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:48.093340 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 13690
I20260810 18:10:48.093497 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:48.093835 22233 external_mini_cluster.cc:1428] Setting key 70f4916db3181c933d7f2f307460ce9f
I20260810 18:10:48.095176 13777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:48.102159 13777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609: Bootstrap starting.
I20260810 18:10:48.103736 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:48.095468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:60621 (local address 127.0.0.1:39729)
0810 18:10:48.096007 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:48.096024 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:48.096063 (+ 39us) server_negotiation.cc:407] Connection header received
0810 18:10:48.096718 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:48.096734 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:48.097123 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:48.097422 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:48.097855 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:48.098894 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:48.100187 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:48.100675 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:48.100800 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:48.101813 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:48.101820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:48.101836 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:48.102281 (+ 445us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:48.102383 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:48.102413 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:48.102523 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:48.102740 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 18:10:48.102916 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":145,"threads_started":1}
I20260810 18:10:48.104924 13777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:48.105860 13777 log.cc:824] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:48.107128 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:48.108371 13777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609: No bootstrap required, opened a new log
I20260810 18:10:48.112494 13777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252c69b9027845659eb429852faa8609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260810 18:10:48.112869 13777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:48.112983 13777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252c69b9027845659eb429852faa8609, State: Initialized, Role: FOLLOWER
I20260810 18:10:48.113492 13777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252c69b9027845659eb429852faa8609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260810 18:10:48.113657 13777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:48.113736 13777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:48.113893 13777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:48.114264 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:48.109672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:56055 (local address 127.0.0.1:39729)
0810 18:10:48.109888 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:48.109893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:48.110123 (+ 230us) server_negotiation.cc:407] Connection header received
0810 18:10:48.110284 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:48.110287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:48.110341 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:48.110432 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:48.110726 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:48.111277 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:48.112204 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:48.112444 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:48.112560 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:48.113605 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:48.113608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:48.113611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:48.113628 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:48.113762 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:48.113785 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:48.113889 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:48.114075 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 18:10:48.114123 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260810 18:10:48.115190 13777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252c69b9027845659eb429852faa8609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260810 18:10:48.115573 13777 leader_election.cc:302] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 252c69b9027845659eb429852faa8609; no voters:
I20260810 18:10:48.115958 13777 leader_election.cc:288] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:48.116165 13789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:48.116505 13789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [term 1 LEADER]: Becoming Leader. State: Replica: 252c69b9027845659eb429852faa8609, State: Running, Role: LEADER
I20260810 18:10:48.116873 13789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252c69b9027845659eb429852faa8609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260810 18:10:48.117321 13777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:48.118779 13791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 252c69b9027845659eb429852faa8609. Latest consensus state: current_term: 1 leader_uuid: "252c69b9027845659eb429852faa8609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252c69b9027845659eb429852faa8609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } } }
I20260810 18:10:48.118918 13791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:48.119328 13790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "252c69b9027845659eb429852faa8609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252c69b9027845659eb429852faa8609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } } }
I20260810 18:10:48.119443 13790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609 [sys.catalog]: This master's current role is: LEADER
W20260810 18:10:48.122884 13804 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:10:48.122989 13804 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:10:48.123081 13801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:48.123843 13801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:48.129978 13801 catalog_manager.cc:1347] Generated new cluster ID: 5a1ac58ef94c487a9ef61a3841370607
I20260810 18:10:48.130057 13801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:48.149641 13801 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:48.150944 13801 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:48.170274 13801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 252c69b9027845659eb429852faa8609: Generated new TSK 0
I20260810 18:10:48.171332 13801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:48.185276 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 13690
W20260810 18:10:48.195353 13672 connection.cc:535] client connection to 127.0.0.1:39729 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39729): Connection reset by peer (error 104)
2026-08-10T18:10:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1568 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:48 dist-test-slave-27sr krb5kdc[13811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:48 dist-test-slave-27sr krb5kdc[13811](info): set up 2 sockets
Aug 10 18:10:48 dist-test-slave-27sr krb5kdc[13811](info): commencing operation
krb5kdc: starting...
W20260810 18:10:49.345057 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:49 dist-test-slave-27sr krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:10:49.484264 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:49.603555 13826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:49.603888 13826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:49.603998 13826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:49.608091 13826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:49.608196 13826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:49.608235 13826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:49.608292 13826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:49.608382 13826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:49.613868 13826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:49.615031 13826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:49.616132 13826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:49.625336 13833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:49.625365 13834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:49.625344 13826 server_base.cc:1034] running on GCE node
W20260810 18:10:49.625347 13836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:49.626260 13826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:49.627449 13826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:49.628646 13826 hybrid_clock.cc:648] HybridClock initialized: now 1786385449628633 us; error 28 us; skew 500 ppm
Aug 10 18:10:49 dist-test-slave-27sr krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:49.636181 13826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:10:49.637265 13826 webserver.cc:460] Webserver started at http://127.0.0.1:33501/ using document root <none> and password file <none>
I20260810 18:10:49.637918 13826 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:49.638022 13826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:49.638288 13826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:49.640805 13826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "2fa41a56e71a40d19671e7be13b1a0e0"
format_stamp: "Formatted at 2026-08-10 18:10:49 on dist-test-slave-27sr"
server_key: "5c3a2d7a4240549d9b22c84a8c6a8fdb"
server_key_iv: "89b0eabcf00e337b6e761060a8f479bd"
server_key_version: "encryptionkey@0"
I20260810 18:10:49.641498 13826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "2fa41a56e71a40d19671e7be13b1a0e0"
format_stamp: "Formatted at 2026-08-10 18:10:49 on dist-test-slave-27sr"
server_key: "5c3a2d7a4240549d9b22c84a8c6a8fdb"
server_key_iv: "89b0eabcf00e337b6e761060a8f479bd"
server_key_version: "encryptionkey@0"
I20260810 18:10:49.646454 13826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 18:10:49.649376 13844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:49.650707 13826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:10:49.650873 13826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "2fa41a56e71a40d19671e7be13b1a0e0"
format_stamp: "Formatted at 2026-08-10 18:10:49 on dist-test-slave-27sr"
server_key: "5c3a2d7a4240549d9b22c84a8c6a8fdb"
server_key_iv: "89b0eabcf00e337b6e761060a8f479bd"
server_key_version: "encryptionkey@0"
I20260810 18:10:49.651010 13826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:49.656489 13826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:49.660080 13826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:49.660286 13826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:49.667619 13826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39465
I20260810 18:10:49.667703 13905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39465 every 8 connection(s)
I20260810 18:10:49.668690 13826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:49.671716 13906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:49.672169 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 13826
I20260810 18:10:49.672334 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:49.672680 22233 external_mini_cluster.cc:1428] Setting key 76100750686a7eb7b108e260a640a5f1
I20260810 18:10:49.678514 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:49.674245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:39955 (local address 127.0.0.1:39465)
0810 18:10:49.674713 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:49.674730 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:49.674768 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:10:49.675472 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:49.675491 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:49.675800 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:49.676187 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:49.677104 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:49.677109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:49.677122 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:49.677133 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:49.677191 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:49.677211 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:49.677328 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:49.677449 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 18:10:49.677655 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":151,"threads_started":1}
I20260810 18:10:49.679615 13906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0: Bootstrap starting.
I20260810 18:10:49.681841 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:49.682549 13906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:49.683537 13906 log.cc:824] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:49.685757 13906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0: No bootstrap required, opened a new log
I20260810 18:10:49.687511 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:49.684385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:36251 (local address 127.0.0.1:39465)
0810 18:10:49.684530 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:49.684535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:49.686159 (+ 1624us) server_negotiation.cc:407] Connection header received
0810 18:10:49.686240 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:49.686243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:49.686279 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:49.686373 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:49.687118 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:49.687121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:49.687124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:49.687136 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:49.687168 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:49.687181 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:49.687273 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:49.687343 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 18:10:49.687384 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:10:49.690090 13906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa41a56e71a40d19671e7be13b1a0e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39465 } }
I20260810 18:10:49.690313 13906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:49.690357 13906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa41a56e71a40d19671e7be13b1a0e0, State: Initialized, Role: FOLLOWER
I20260810 18:10:49.690781 13906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa41a56e71a40d19671e7be13b1a0e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39465 } }
I20260810 18:10:49.690907 13906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:49.690955 13906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:49.691027 13906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:49.692003 13906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa41a56e71a40d19671e7be13b1a0e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39465 } }
I20260810 18:10:49.692310 13906 leader_election.cc:302] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fa41a56e71a40d19671e7be13b1a0e0; no voters:
I20260810 18:10:49.692579 13906 leader_election.cc:288] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:49.692713 13919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:49.693005 13919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [term 1 LEADER]: Becoming Leader. State: Replica: 2fa41a56e71a40d19671e7be13b1a0e0, State: Running, Role: LEADER
I20260810 18:10:49.693373 13919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa41a56e71a40d19671e7be13b1a0e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39465 } }
I20260810 18:10:49.693984 13906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:49.695153 13921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fa41a56e71a40d19671e7be13b1a0e0. Latest consensus state: current_term: 1 leader_uuid: "2fa41a56e71a40d19671e7be13b1a0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa41a56e71a40d19671e7be13b1a0e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39465 } } }
I20260810 18:10:49.695312 13921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:49.695194 13920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fa41a56e71a40d19671e7be13b1a0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa41a56e71a40d19671e7be13b1a0e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39465 } } }
I20260810 18:10:49.695650 13920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:49.695739 13931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:49.699396 13931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:49.705644 13931 catalog_manager.cc:1347] Generated new cluster ID: f6726ae81ecc4ddf8f980963abde618b
I20260810 18:10:49.705735 13931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:49.714700 13931 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:49.715045 13931 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:49.723356 13931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fa41a56e71a40d19671e7be13b1a0e0: Generated new TSK 0
I20260810 18:10:49.724108 13931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:49.733376 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 13826
2026-08-10T18:10:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1546 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:49 dist-test-slave-27sr krb5kdc[13946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:49 dist-test-slave-27sr krb5kdc[13946](info): set up 2 sockets
Aug 10 18:10:49 dist-test-slave-27sr krb5kdc[13946](info): commencing operation
krb5kdc: starting...
W20260810 18:10:50.884840 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:50 dist-test-slave-27sr krb5kdc[13946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:10:51.025422 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45021
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:51.145490 13961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:51.145850 13961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:51.145946 13961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:51.150199 13961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:51.150314 13961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:51.150353 13961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:51.150395 13961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:51.150432 13961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:51.155929 13961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45021
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:51.157158 13961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:51.158291 13961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:51.167677 13967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:51.167619 13966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:51.167737 13961 server_base.cc:1034] running on GCE node
W20260810 18:10:51.167619 13970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:51.168635 13961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:51.169822 13961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:51.171041 13961 hybrid_clock.cc:648] HybridClock initialized: now 1786385451171024 us; error 31 us; skew 500 ppm
Aug 10 18:10:51 dist-test-slave-27sr krb5kdc[13946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:51.174064 13961 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:10:51.175100 13961 webserver.cc:460] Webserver started at http://127.0.0.1:40753/ using document root <none> and password file <none>
I20260810 18:10:51.175673 13961 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:51.175761 13961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:51.176119 13961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:51.178359 13961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "5cc1d7e11d83440faf2f76fb0af9ec2b"
format_stamp: "Formatted at 2026-08-10 18:10:51 on dist-test-slave-27sr"
server_key: "ee6df7cbb1710d2b958850d41e1ce236"
server_key_iv: "4d48b985944bd3bb4273ea8d6975e0ea"
server_key_version: "encryptionkey@0"
I20260810 18:10:51.178947 13961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "5cc1d7e11d83440faf2f76fb0af9ec2b"
format_stamp: "Formatted at 2026-08-10 18:10:51 on dist-test-slave-27sr"
server_key: "ee6df7cbb1710d2b958850d41e1ce236"
server_key_iv: "4d48b985944bd3bb4273ea8d6975e0ea"
server_key_version: "encryptionkey@0"
I20260810 18:10:51.183382 13961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:10:51.186195 13977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:51.187610 13961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:10:51.187783 13961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "5cc1d7e11d83440faf2f76fb0af9ec2b"
format_stamp: "Formatted at 2026-08-10 18:10:51 on dist-test-slave-27sr"
server_key: "ee6df7cbb1710d2b958850d41e1ce236"
server_key_iv: "4d48b985944bd3bb4273ea8d6975e0ea"
server_key_version: "encryptionkey@0"
I20260810 18:10:51.187950 13961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:51.239292 13961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:51.242714 13961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:51.242967 13961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:51.250794 13961 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45021
I20260810 18:10:51.250808 14040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45021 every 8 connection(s)
I20260810 18:10:51.251870 13961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:51.254076 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 13961
I20260810 18:10:51.254251 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:51.254602 22233 external_mini_cluster.cc:1428] Setting key c447dde19b5b2701bfa27afe3436c81c
I20260810 18:10:51.255430 14041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:51.263037 14041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b: Bootstrap starting.
Aug 10 18:10:51 dist-test-slave-27sr krb5kdc[13946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 18:10:51.266062 14041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:51.266861 14041 log.cc:824] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:51.269531 14041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b: No bootstrap required, opened a new log
I20260810 18:10:51.270953 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:51.256215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:48003 (local address 127.0.0.1:45021)
0810 18:10:51.256665 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:51.256692 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:51.256731 (+ 39us) server_negotiation.cc:407] Connection header received
0810 18:10:51.257454 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:51.257473 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:51.257876 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:51.258182 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:51.259228 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:51.260368 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:51.261638 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:51.262132 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:51.262252 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:51.265377 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:51.265382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:51.265399 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:51.265434 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:51.267599 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:51.268306 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:51.268313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:51.268318 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:51.268448 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:51.268843 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:51.268846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:51.268847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:51.269266 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:51.269434 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:51.269880 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 18:10:51.270173 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":144,"threads_started":1}
I20260810 18:10:51.274668 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:51.274430 14041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc1d7e11d83440faf2f76fb0af9ec2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45021 } }
I20260810 18:10:51.274788 14041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:51.274863 14041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc1d7e11d83440faf2f76fb0af9ec2b, State: Initialized, Role: FOLLOWER
I20260810 18:10:51.275332 14041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc1d7e11d83440faf2f76fb0af9ec2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45021 } }
I20260810 18:10:51.275465 14041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:51.275555 14041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:51.275664 14041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:51.276947 14041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc1d7e11d83440faf2f76fb0af9ec2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45021 } }
I20260810 18:10:51.277424 14041 leader_election.cc:302] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cc1d7e11d83440faf2f76fb0af9ec2b; no voters:
I20260810 18:10:51.277848 14041 leader_election.cc:288] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:51.277981 14053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:51.278247 14053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [term 1 LEADER]: Becoming Leader. State: Replica: 5cc1d7e11d83440faf2f76fb0af9ec2b, State: Running, Role: LEADER
I20260810 18:10:51.278729 14053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc1d7e11d83440faf2f76fb0af9ec2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45021 } }
I20260810 18:10:51.279469 14041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:51.280890 14055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cc1d7e11d83440faf2f76fb0af9ec2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc1d7e11d83440faf2f76fb0af9ec2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45021 } } }
I20260810 18:10:51.280897 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cc1d7e11d83440faf2f76fb0af9ec2b. Latest consensus state: current_term: 1 leader_uuid: "5cc1d7e11d83440faf2f76fb0af9ec2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc1d7e11d83440faf2f76fb0af9ec2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45021 } } }
I20260810 18:10:51.281073 14055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:51.281080 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:51.281520 14065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:51.285374 14065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:51.289378 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:51.277251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:44653 (local address 127.0.0.1:45021)
0810 18:10:51.277424 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:51.277429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:51.277671 (+ 242us) server_negotiation.cc:407] Connection header received
0810 18:10:51.277878 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:51.277882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:51.277942 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:51.278035 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:51.279144 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:51.279989 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:51.282302 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:51.282549 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:51.282678 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:51.284620 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:51.284624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:51.284626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:51.284656 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:51.286540 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:51.287205 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:51.287209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:51.287212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:51.287323 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:51.287766 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:51.287771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:51.287773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:51.288067 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:51.288225 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:51.289088 (+ 863us) server_negotiation.cc:299] Negotiation successful
0810 18:10:51.289207 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:10:51.292214 14065 catalog_manager.cc:1347] Generated new cluster ID: ceedffdb4a774c8580f23c701999ce28
I20260810 18:10:51.292326 14065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:51.309150 14065 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:51.310302 14065 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:51.320501 14065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cc1d7e11d83440faf2f76fb0af9ec2b: Generated new TSK 0
I20260810 18:10:51.321440 14065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:51.327246 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 13961
W20260810 18:10:51.338922 13942 connection.cc:535] client connection to 127.0.0.1:45021 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45021): Connection reset by peer (error 104)
2026-08-10T18:10:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1597 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:51 dist-test-slave-27sr krb5kdc[14077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:51 dist-test-slave-27sr krb5kdc[14077](info): set up 2 sockets
Aug 10 18:10:51 dist-test-slave-27sr krb5kdc[14077](info): commencing operation
krb5kdc: starting...
W20260810 18:10:52.475229 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:52 dist-test-slave-27sr krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:10:52.622748 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:52.745132 14091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:52.745497 14091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:52.745590 14091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:52.749898 14091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:52.750026 14091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:52.750066 14091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:52.750108 14091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:52.750144 14091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:52.755754 14091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43719
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:52.757094 14091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:52.758302 14091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:10:52.768442 14091 server_base.cc:1034] running on GCE node
W20260810 18:10:52.768368 14096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:52.768333 14099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:52.768358 14097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:52.769428 14091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:52.770787 14091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:52.771989 14091 hybrid_clock.cc:648] HybridClock initialized: now 1786385452771943 us; error 53 us; skew 500 ppm
Aug 10 18:10:52 dist-test-slave-27sr krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:52.775094 14091 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:10:52.776208 14091 webserver.cc:460] Webserver started at http://127.0.0.1:42657/ using document root <none> and password file <none>
I20260810 18:10:52.776860 14091 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:52.776973 14091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:52.777241 14091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:52.779508 14091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "a1241aff56574922be0b41e5389dd2fc"
format_stamp: "Formatted at 2026-08-10 18:10:52 on dist-test-slave-27sr"
server_key: "324edf32ac85f723ce666c1d49671beb"
server_key_iv: "9ea759dbf0419f8153efc6ae10a6bcda"
server_key_version: "encryptionkey@0"
I20260810 18:10:52.780194 14091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "a1241aff56574922be0b41e5389dd2fc"
format_stamp: "Formatted at 2026-08-10 18:10:52 on dist-test-slave-27sr"
server_key: "324edf32ac85f723ce666c1d49671beb"
server_key_iv: "9ea759dbf0419f8153efc6ae10a6bcda"
server_key_version: "encryptionkey@0"
I20260810 18:10:52.784473 14091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:10:52.787417 14109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:52.788957 14091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:52.789243 14091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "a1241aff56574922be0b41e5389dd2fc"
format_stamp: "Formatted at 2026-08-10 18:10:52 on dist-test-slave-27sr"
server_key: "324edf32ac85f723ce666c1d49671beb"
server_key_iv: "9ea759dbf0419f8153efc6ae10a6bcda"
server_key_version: "encryptionkey@0"
I20260810 18:10:52.789394 14091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:52.802614 14091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:52.806098 14091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:52.806346 14091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:52.814703 14091 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43719
I20260810 18:10:52.814708 14177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43719 every 8 connection(s)
I20260810 18:10:52.815989 14091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:52.819180 14182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:52.821528 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 14091
I20260810 18:10:52.821710 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:52.822018 22233 external_mini_cluster.cc:1428] Setting key 1864f51886afdd09e44c4637634d31c1
I20260810 18:10:52.826696 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc: Bootstrap starting.
I20260810 18:10:52.829696 14182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:52.830701 14182 log.cc:824] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:52.833485 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc: No bootstrap required, opened a new log
I20260810 18:10:52.833912 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:52.823675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:55751 (local address 127.0.0.1:43719)
0810 18:10:52.824198 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:52.824219 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:52.824265 (+ 46us) server_negotiation.cc:407] Connection header received
0810 18:10:52.825414 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:52.825442 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:52.825945 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:52.826359 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:52.826822 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:52.828388 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:52.829501 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:52.830258 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:52.830429 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:52.831503 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:52.831509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:52.831525 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:52.832240 (+ 715us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:52.832318 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:52.832351 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:52.832514 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:52.832717 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 18:10:52.832876 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":176,"threads_started":1}
I20260810 18:10:52.837618 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:52.838315 14182 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1241aff56574922be0b41e5389dd2fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43719 } }
I20260810 18:10:52.838565 14182 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:52.838635 14182 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1241aff56574922be0b41e5389dd2fc, State: Initialized, Role: FOLLOWER
I20260810 18:10:52.839211 14182 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1241aff56574922be0b41e5389dd2fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43719 } }
I20260810 18:10:52.839356 14182 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:52.839447 14182 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:52.839576 14182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:52.841079 14182 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1241aff56574922be0b41e5389dd2fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43719 } }
I20260810 18:10:52.841531 14182 leader_election.cc:302] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1241aff56574922be0b41e5389dd2fc; no voters:
I20260810 18:10:52.841975 14182 leader_election.cc:288] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:52.842204 14194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:52.842597 14194 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [term 1 LEADER]: Becoming Leader. State: Replica: a1241aff56574922be0b41e5389dd2fc, State: Running, Role: LEADER
I20260810 18:10:52.843873 14194 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1241aff56574922be0b41e5389dd2fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43719 } }
I20260810 18:10:52.844063 14182 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:52.846920 14195 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1241aff56574922be0b41e5389dd2fc. Latest consensus state: current_term: 1 leader_uuid: "a1241aff56574922be0b41e5389dd2fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1241aff56574922be0b41e5389dd2fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43719 } } }
I20260810 18:10:52.847086 14195 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:52.847100 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:52.840115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:54527 (local address 127.0.0.1:43719)
0810 18:10:52.840246 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:52.840255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:52.840479 (+ 224us) server_negotiation.cc:407] Connection header received
0810 18:10:52.840610 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:52.840612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:52.840659 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:52.840731 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:52.841087 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:52.841960 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:52.844368 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:52.844730 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:52.844841 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:52.846070 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:52.846077 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:52.846082 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:52.846105 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:52.846146 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:52.846167 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:52.846278 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:52.846873 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 18:10:52.846922 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:10:52.847718 14196 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1241aff56574922be0b41e5389dd2fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1241aff56574922be0b41e5389dd2fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43719 } } }
I20260810 18:10:52.848042 14196 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:52.848924 14210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:52.851739 14210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:52.858127 14210 catalog_manager.cc:1347] Generated new cluster ID: 9ba710751ea74901841d2364bc91ef35
I20260810 18:10:52.858242 14210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:52.870107 14210 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:52.871275 14210 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:52.881292 14210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1241aff56574922be0b41e5389dd2fc: Generated new TSK 0
I20260810 18:10:52.882153 14210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:52.890810 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 14091
W20260810 18:10:52.900302 14075 connection.cc:535] client connection to 127.0.0.1:43719 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43719): Connection reset by peer (error 104)
2026-08-10T18:10:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1560 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:52 dist-test-slave-27sr krb5kdc[14216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:52 dist-test-slave-27sr krb5kdc[14216](info): set up 2 sockets
Aug 10 18:10:52 dist-test-slave-27sr krb5kdc[14216](info): commencing operation
krb5kdc: starting...
W20260810 18:10:54.027493 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:54 dist-test-slave-27sr krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 18:10:54.167824 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:54.286644 14235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:54.286964 14235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:54.287050 14235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:54.291440 14235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:54.291532 14235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:54.291553 14235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:54.291596 14235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:54.291635 14235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:54.297333 14235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44989
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:54.298624 14235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:54.299804 14235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:54.309568 14244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:54.309628 14235 server_base.cc:1034] running on GCE node
W20260810 18:10:54.309650 14241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:54.309568 14242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:54.310541 14235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:54.311816 14235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:54.313035 14235 hybrid_clock.cc:648] HybridClock initialized: now 1786385454313012 us; error 35 us; skew 500 ppm
Aug 10 18:10:54 dist-test-slave-27sr krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:54.316191 14235 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 18:10:54.317142 14235 webserver.cc:460] Webserver started at http://127.0.0.1:35281/ using document root <none> and password file <none>
I20260810 18:10:54.317713 14235 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:54.317775 14235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:54.317947 14235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:54.320125 14235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "2ac466c1f360417eab1c664ec0b10ba5"
format_stamp: "Formatted at 2026-08-10 18:10:54 on dist-test-slave-27sr"
server_key: "1524a2fcfc8a679f5cd10fac67ce0666"
server_key_iv: "f22def7504dbe21bef5ef2bda92c3b17"
server_key_version: "encryptionkey@0"
I20260810 18:10:54.320685 14235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "2ac466c1f360417eab1c664ec0b10ba5"
format_stamp: "Formatted at 2026-08-10 18:10:54 on dist-test-slave-27sr"
server_key: "1524a2fcfc8a679f5cd10fac67ce0666"
server_key_iv: "f22def7504dbe21bef5ef2bda92c3b17"
server_key_version: "encryptionkey@0"
I20260810 18:10:54.324635 14235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:10:54.327123 14251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:54.328440 14235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:10:54.328548 14235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "2ac466c1f360417eab1c664ec0b10ba5"
format_stamp: "Formatted at 2026-08-10 18:10:54 on dist-test-slave-27sr"
server_key: "1524a2fcfc8a679f5cd10fac67ce0666"
server_key_iv: "f22def7504dbe21bef5ef2bda92c3b17"
server_key_version: "encryptionkey@0"
I20260810 18:10:54.328634 14235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:54.333346 14235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:54.336526 14235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:54.336686 14235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:54.343688 14235 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44989
I20260810 18:10:54.343801 14314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44989 every 8 connection(s)
I20260810 18:10:54.344753 14235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:54.345773 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 14235
I20260810 18:10:54.345933 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:54.346273 22233 external_mini_cluster.cc:1428] Setting key 3f0e88d6d6a04db576fb25864de42c4c
I20260810 18:10:54.348547 14315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:54.352156 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:54.347708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:41711 (local address 127.0.0.1:44989)
0810 18:10:54.348215 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:54.348237 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:54.348283 (+ 46us) server_negotiation.cc:407] Connection header received
0810 18:10:54.349150 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:54.349174 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:54.349527 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:54.349900 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:54.350701 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:54.350707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:54.350723 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:54.350743 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:54.350813 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:54.350839 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:54.350985 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:54.351052 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 18:10:54.351209 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":191,"threads_started":1}
I20260810 18:10:54.355075 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:54.355594 14315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5: Bootstrap starting.
I20260810 18:10:54.358106 14315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:54.358922 14315 log.cc:824] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:54.359095 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:54.357267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:53327 (local address 127.0.0.1:44989)
0810 18:10:54.357405 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:54.357410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:54.357581 (+ 171us) server_negotiation.cc:407] Connection header received
0810 18:10:54.357746 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:54.357748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:54.357778 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:54.357860 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:54.358605 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:54.358608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:54.358610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:54.358623 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:54.358666 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:54.358678 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:54.358768 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:54.358861 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 18:10:54.358903 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:10:54.361058 14315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5: No bootstrap required, opened a new log
I20260810 18:10:54.364547 14315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac466c1f360417eab1c664ec0b10ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260810 18:10:54.364777 14315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:54.364863 14315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac466c1f360417eab1c664ec0b10ba5, State: Initialized, Role: FOLLOWER
I20260810 18:10:54.365352 14315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac466c1f360417eab1c664ec0b10ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260810 18:10:54.365502 14315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:54.365571 14315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:54.365679 14315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:54.366611 14315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac466c1f360417eab1c664ec0b10ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260810 18:10:54.366971 14315 leader_election.cc:302] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ac466c1f360417eab1c664ec0b10ba5; no voters:
I20260810 18:10:54.367408 14315 leader_election.cc:288] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:54.367443 14327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:54.367633 14327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [term 1 LEADER]: Becoming Leader. State: Replica: 2ac466c1f360417eab1c664ec0b10ba5, State: Running, Role: LEADER
I20260810 18:10:54.368048 14327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac466c1f360417eab1c664ec0b10ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260810 18:10:54.368798 14315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:54.370105 14330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ac466c1f360417eab1c664ec0b10ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac466c1f360417eab1c664ec0b10ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } } }
I20260810 18:10:54.370117 14328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ac466c1f360417eab1c664ec0b10ba5. Latest consensus state: current_term: 1 leader_uuid: "2ac466c1f360417eab1c664ec0b10ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ac466c1f360417eab1c664ec0b10ba5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } } }
I20260810 18:10:54.370282 14330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:54.370296 14328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:54.370697 14339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:54.374184 14339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:54.380151 14339 catalog_manager.cc:1347] Generated new cluster ID: c5c1f14017894e12ab9ff3e98ecb621e
I20260810 18:10:54.380239 14339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:54.404973 14339 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:54.405335 14339 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:54.413141 14339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ac466c1f360417eab1c664ec0b10ba5: Generated new TSK 0
I20260810 18:10:54.413846 14339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:54.421622 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 14235
2026-08-10T18:10:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1531 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:54 dist-test-slave-27sr krb5kdc[14353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:54 dist-test-slave-27sr krb5kdc[14353](info): set up 2 sockets
Aug 10 18:10:54 dist-test-slave-27sr krb5kdc[14353](info): commencing operation
krb5kdc: starting...
W20260810 18:10:55.559338 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:55 dist-test-slave-27sr krb5kdc[14353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:10:55.704099 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:43993
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:45541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:43993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:55.828585 14369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:55.828974 14369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:55.829071 14369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:55.833514 14369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:55.833621 14369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:55.833657 14369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:55.833716 14369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:55.833777 14369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:55.839645 14369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:45541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:43993
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:43993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:55.840974 14369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:55.842170 14369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:10:55.851812 14369 server_base.cc:1034] running on GCE node
W20260810 18:10:55.851794 14378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:55.851796 14375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:55.852034 14376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:55.852834 14369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:55.854305 14369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:55.855531 14369 hybrid_clock.cc:648] HybridClock initialized: now 1786385455855493 us; error 58 us; skew 500 ppm
Aug 10 18:10:55 dist-test-slave-27sr krb5kdc[14353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385455, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:55.862169 14369 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:10:55.863354 14369 webserver.cc:460] Webserver started at http://127.21.182.126:42681/ using document root <none> and password file <none>
I20260810 18:10:55.864073 14369 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:55.864148 14369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:55.864405 14369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:55.866777 14369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "5079e6b9e57f4d009d8fdb7377822858"
format_stamp: "Formatted at 2026-08-10 18:10:55 on dist-test-slave-27sr"
server_key: "f35ccbafafe1d8990109913003cd8dd4"
server_key_iv: "66b5ab2894e64fad95f8f17269eca5f5"
server_key_version: "encryptionkey@0"
I20260810 18:10:55.867434 14369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "5079e6b9e57f4d009d8fdb7377822858"
format_stamp: "Formatted at 2026-08-10 18:10:55 on dist-test-slave-27sr"
server_key: "f35ccbafafe1d8990109913003cd8dd4"
server_key_iv: "66b5ab2894e64fad95f8f17269eca5f5"
server_key_version: "encryptionkey@0"
I20260810 18:10:55.872020 14369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 18:10:55.874975 14385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:55.876585 14369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:10:55.876833 14369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "5079e6b9e57f4d009d8fdb7377822858"
format_stamp: "Formatted at 2026-08-10 18:10:55 on dist-test-slave-27sr"
server_key: "f35ccbafafe1d8990109913003cd8dd4"
server_key_iv: "66b5ab2894e64fad95f8f17269eca5f5"
server_key_version: "encryptionkey@0"
I20260810 18:10:55.876958 14369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:55.893168 14369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:55.897168 14369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:55.897414 14369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:55.905857 14448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:43993 every 8 connection(s)
I20260810 18:10:55.905927 14369 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:43993
I20260810 18:10:55.907076 14369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:55.910676 14449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:55.912997 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 14369
I20260810 18:10:55.913189 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:55.913542 22233 external_mini_cluster.cc:1428] Setting key d976e18585cbf2b32b23bb1a29e7a7fe
I20260810 18:10:55.918672 14449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858: Bootstrap starting.
I20260810 18:10:55.921422 14449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:55.922271 14449 log.cc:824] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:10:55 dist-test-slave-27sr krb5kdc[14353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:10:55.924803 14449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858: No bootstrap required, opened a new log
I20260810 18:10:55.928992 14449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5079e6b9e57f4d009d8fdb7377822858" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43993 } }
I20260810 18:10:55.929397 14449 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:55.929510 14449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5079e6b9e57f4d009d8fdb7377822858, State: Initialized, Role: FOLLOWER
I20260810 18:10:55.930090 14449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5079e6b9e57f4d009d8fdb7377822858" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43993 } }
I20260810 18:10:55.930244 14449 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:55.930356 14449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:55.930438 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:55.915338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34445 (local address 127.21.182.126:43993)
0810 18:10:55.915949 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:55.915967 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:55.916007 (+ 40us) server_negotiation.cc:407] Connection header received
0810 18:10:55.916694 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:55.916712 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:55.917089 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:55.917467 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:55.918343 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:55.919453 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:55.920447 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:55.920954 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:55.921127 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:55.924380 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:55.924386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:55.924403 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:55.924436 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:55.926539 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:55.927175 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:55.927181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:55.927187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:55.927316 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:55.927739 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:55.927742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:55.927744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:55.928293 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:55.928570 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:55.928942 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 18:10:55.929217 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":168,"threads_started":1}
I20260810 18:10:55.930495 14449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:55.931800 14449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5079e6b9e57f4d009d8fdb7377822858" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43993 } }
I20260810 18:10:55.932256 14449 leader_election.cc:302] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5079e6b9e57f4d009d8fdb7377822858; no voters:
I20260810 18:10:55.932684 14449 leader_election.cc:288] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:55.932780 14455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:55.933017 14455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [term 1 LEADER]: Becoming Leader. State: Replica: 5079e6b9e57f4d009d8fdb7377822858, State: Running, Role: LEADER
I20260810 18:10:55.933564 14455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5079e6b9e57f4d009d8fdb7377822858" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43993 } }
I20260810 18:10:55.934309 14449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:55.934654 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:55.935022 14455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5079e6b9e57f4d009d8fdb7377822858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5079e6b9e57f4d009d8fdb7377822858" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43993 } } }
I20260810 18:10:55.935256 14455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:55.935527 14456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5079e6b9e57f4d009d8fdb7377822858. Latest consensus state: current_term: 1 leader_uuid: "5079e6b9e57f4d009d8fdb7377822858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5079e6b9e57f4d009d8fdb7377822858" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 43993 } } }
I20260810 18:10:55.935667 14456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:55.935772 14465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:55.939983 14465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:55.947885 14465 catalog_manager.cc:1347] Generated new cluster ID: 4d829a866c65415f96c7ed40c936ad61
I20260810 18:10:55.947991 14465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:55.948984 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:55.937429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55083 (local address 127.21.182.126:43993)
0810 18:10:55.937561 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:55.937567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:55.938104 (+ 537us) server_negotiation.cc:407] Connection header received
0810 18:10:55.938322 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:55.938327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:55.938386 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:55.938469 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:55.939625 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:55.940560 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:55.941539 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:55.941838 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:55.941971 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:55.943988 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:55.943993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:55.943997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:10:55.944029 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:55.946240 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:55.946887 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:55.946890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:55.946892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:55.946980 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:10:55.947394 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:10:55.947397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:10:55.947399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:10:55.947695 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:55.947873 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:55.948691 (+ 818us) server_negotiation.cc:299] Negotiation successful
0810 18:10:55.948819 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:10:55.962968 14465 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:55.964541 14465 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:55.975426 14465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5079e6b9e57f4d009d8fdb7377822858: Generated new TSK 0
I20260810 18:10:55.976320 14465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:55.984349 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 14369
W20260810 18:10:55.995076 14351 connection.cc:535] client connection to 127.21.182.126:43993 recv error: Network error: failed to read from TLS socket (remote: 127.21.182.126:43993): Connection reset by peer (error 104)
2026-08-10T18:10:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1562 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:56 dist-test-slave-27sr krb5kdc[14485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:56 dist-test-slave-27sr krb5kdc[14485](info): set up 2 sockets
Aug 10 18:10:56 dist-test-slave-27sr krb5kdc[14485](info): commencing operation
krb5kdc: starting...
W20260810 18:10:57.138514 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:57 dist-test-slave-27sr krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:10:57.287756 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:34795
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:34795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:57.409404 14499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:57.409765 14499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:57.409861 14499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:57.414119 14499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:57.414250 14499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:57.414291 14499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:57.414330 14499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:57.414379 14499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:57.420260 14499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:34795
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:34795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:57.421545 14499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:57.422698 14499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:57.432538 14508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:57.432688 14499 server_base.cc:1034] running on GCE node
W20260810 18:10:57.432546 14505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:57.432546 14506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:57.433560 14499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:57.434891 14499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:57.436096 14499 hybrid_clock.cc:648] HybridClock initialized: now 1786385457436059 us; error 54 us; skew 500 ppm
Aug 10 18:10:57 dist-test-slave-27sr krb5kdc[14485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385457, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:57.439401 14499 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:10:57.440555 14499 webserver.cc:460] Webserver started at http://127.21.182.126:33111/ using document root <none> and password file <none>
I20260810 18:10:57.441201 14499 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:57.441339 14499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:57.441608 14499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:57.444000 14499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "3257932d40d14814b031a0863d1ae80a"
format_stamp: "Formatted at 2026-08-10 18:10:57 on dist-test-slave-27sr"
server_key: "6e439385b17c3001a0f6efa0fa21bbf6"
server_key_iv: "8535f92e2780e00fc4f6799fc149ee91"
server_key_version: "encryptionkey@0"
I20260810 18:10:57.444630 14499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "3257932d40d14814b031a0863d1ae80a"
format_stamp: "Formatted at 2026-08-10 18:10:57 on dist-test-slave-27sr"
server_key: "6e439385b17c3001a0f6efa0fa21bbf6"
server_key_iv: "8535f92e2780e00fc4f6799fc149ee91"
server_key_version: "encryptionkey@0"
I20260810 18:10:57.448745 14499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:10:57.451308 14516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:57.452644 14499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:10:57.452791 14499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "3257932d40d14814b031a0863d1ae80a"
format_stamp: "Formatted at 2026-08-10 18:10:57 on dist-test-slave-27sr"
server_key: "6e439385b17c3001a0f6efa0fa21bbf6"
server_key_iv: "8535f92e2780e00fc4f6799fc149ee91"
server_key_version: "encryptionkey@0"
I20260810 18:10:57.452909 14499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:57.480034 14499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:57.483629 14499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:57.483848 14499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:57.491319 14499 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:34795
I20260810 18:10:57.491369 14580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:34795 every 8 connection(s)
I20260810 18:10:57.492395 14499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:57.495313 14581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:57.496474 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 14499
I20260810 18:10:57.496641 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:57.496942 22233 external_mini_cluster.cc:1428] Setting key 4469b9af9b561a2b8adcc58ad00b91dc
I20260810 18:10:57.502401 14581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a: Bootstrap starting.
I20260810 18:10:57.505136 14581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:57.505951 14581 log.cc:824] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:57.507305 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:57.498471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52565 (local address 127.21.182.126:34795)
0810 18:10:57.498927 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:57.498948 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:57.498993 (+ 45us) server_negotiation.cc:407] Connection header received
0810 18:10:57.499824 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:57.499849 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:57.500350 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:57.500724 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:57.501163 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:57.502629 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:57.503540 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:57.504320 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:57.504493 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:57.505409 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:57.505415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:57.505428 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:57.505822 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:57.505902 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:57.505927 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:57.506082 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:57.506257 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 18:10:57.506434 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":122,"threads_started":1}
I20260810 18:10:57.508168 14581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a: No bootstrap required, opened a new log
I20260810 18:10:57.510303 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:57.512183 14581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3257932d40d14814b031a0863d1ae80a" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34795 } }
I20260810 18:10:57.512487 14581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:57.512562 14581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3257932d40d14814b031a0863d1ae80a, State: Initialized, Role: FOLLOWER
I20260810 18:10:57.512984 14581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3257932d40d14814b031a0863d1ae80a" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34795 } }
I20260810 18:10:57.513108 14581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:57.513154 14581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:57.513226 14581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:57.514078 14581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3257932d40d14814b031a0863d1ae80a" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34795 } }
I20260810 18:10:57.514427 14581 leader_election.cc:302] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3257932d40d14814b031a0863d1ae80a; no voters:
I20260810 18:10:57.514758 14581 leader_election.cc:288] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:57.514837 14592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:57.515075 14592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [term 1 LEADER]: Becoming Leader. State: Replica: 3257932d40d14814b031a0863d1ae80a, State: Running, Role: LEADER
I20260810 18:10:57.515446 14592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3257932d40d14814b031a0863d1ae80a" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34795 } }
I20260810 18:10:57.516144 14581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:57.516776 14594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3257932d40d14814b031a0863d1ae80a. Latest consensus state: current_term: 1 leader_uuid: "3257932d40d14814b031a0863d1ae80a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3257932d40d14814b031a0863d1ae80a" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34795 } } }
I20260810 18:10:57.516919 14594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:57.517407 14600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:57.517867 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:57.512502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54473 (local address 127.21.182.126:34795)
0810 18:10:57.512622 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:57.512628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:57.513433 (+ 805us) server_negotiation.cc:407] Connection header received
0810 18:10:57.513568 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:57.513572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:57.513618 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:57.513704 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:57.513979 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:57.514596 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:57.515717 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:10:57.516033 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:10:57.516130 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:10:57.517204 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:57.517209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:57.517213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:57.517226 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:57.517254 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:57.517267 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:57.517353 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:57.517707 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 18:10:57.517745 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:10:57.518362 14593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3257932d40d14814b031a0863d1ae80a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3257932d40d14814b031a0863d1ae80a" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34795 } } }
I20260810 18:10:57.518497 14593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:57.521420 14600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:57.527424 14600 catalog_manager.cc:1347] Generated new cluster ID: 88f5eb1d4c2a4fcf8c0f90619f83d5f9
I20260810 18:10:57.527520 14600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:57.537569 14600 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:57.538601 14600 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:57.545882 14600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3257932d40d14814b031a0863d1ae80a: Generated new TSK 0
I20260810 18:10:57.546593 14600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:57.555498 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 14499
W20260810 18:10:57.564874 14483 connection.cc:535] client connection to 127.21.182.126:34795 recv error: Network error: failed to read from TLS socket (remote: 127.21.182.126:34795): Connection reset by peer (error 104)
2026-08-10T18:10:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1570 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:57 dist-test-slave-27sr krb5kdc[14619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:57 dist-test-slave-27sr krb5kdc[14619](info): set up 2 sockets
Aug 10 18:10:57 dist-test-slave-27sr krb5kdc[14619](info): commencing operation
krb5kdc: starting...
W20260810 18:10:58.752514 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:10:58 dist-test-slave-27sr krb5kdc[14619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:10:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:10:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:10:58.895870 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:45141
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:44279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:45141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:10:59.015450 14634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:10:59.015720 14634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:10:59.015764 14634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:10:59.019945 14634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:10:59.020016 14634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:10:59.020037 14634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:10:59.020056 14634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:10:59.020131 14634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:10:59.026436 14634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:44279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:45141
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:45141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:10:59.027617 14634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:10:59.028767 14634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:10:59.038098 14641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:59.038076 14640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:10:59.038106 14643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:10:59.038388 14634 server_base.cc:1034] running on GCE node
I20260810 18:10:59.039165 14634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:10:59.040464 14634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:10:59.041641 14634 hybrid_clock.cc:648] HybridClock initialized: now 1786385459041620 us; error 32 us; skew 500 ppm
Aug 10 18:10:59 dist-test-slave-27sr krb5kdc[14619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385459, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:10:59.044566 14634 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:10:59.045567 14634 webserver.cc:460] Webserver started at http://127.21.182.126:45607/ using document root <none> and password file <none>
I20260810 18:10:59.046154 14634 fs_manager.cc:359] Metadata directory not provided
I20260810 18:10:59.046249 14634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:10:59.046497 14634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:10:59.048700 14634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "2321594e4ca74308a0bc1b563d177a29"
format_stamp: "Formatted at 2026-08-10 18:10:59 on dist-test-slave-27sr"
server_key: "2eae81ddf9f108855bf1a5354f68c384"
server_key_iv: "3504f7162efea9d09f39b33b7af9b296"
server_key_version: "encryptionkey@0"
I20260810 18:10:59.049320 14634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "2321594e4ca74308a0bc1b563d177a29"
format_stamp: "Formatted at 2026-08-10 18:10:59 on dist-test-slave-27sr"
server_key: "2eae81ddf9f108855bf1a5354f68c384"
server_key_iv: "3504f7162efea9d09f39b33b7af9b296"
server_key_version: "encryptionkey@0"
I20260810 18:10:59.053256 14634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:10:59.055768 14652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:10:59.057054 14634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:10:59.057200 14634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "2321594e4ca74308a0bc1b563d177a29"
format_stamp: "Formatted at 2026-08-10 18:10:59 on dist-test-slave-27sr"
server_key: "2eae81ddf9f108855bf1a5354f68c384"
server_key_iv: "3504f7162efea9d09f39b33b7af9b296"
server_key_version: "encryptionkey@0"
I20260810 18:10:59.057354 14634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:10:59.062104 14634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:10:59.065085 14634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:10:59.065289 14634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:10:59.072571 14634 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:45141
I20260810 18:10:59.072603 14710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:45141 every 8 connection(s)
I20260810 18:10:59.073555 14634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:10:59.076514 14711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:10:59.082536 14711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29: Bootstrap starting.
I20260810 18:10:59.083652 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 14634
I20260810 18:10:59.083949 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:10:59.084405 22233 external_mini_cluster.cc:1428] Setting key 0484abf7d3db22af71db8f1f6542e9ae
I20260810 18:10:59.086087 14711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29: Neither blocks nor log segments found. Creating new log.
I20260810 18:10:59.086934 14711 log.cc:824] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29: Log is configured to *not* fsync() on all Append() calls
I20260810 18:10:59.089776 14711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29: No bootstrap required, opened a new log
I20260810 18:10:59.090363 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:59.085781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33049 (local address 127.21.182.126:45141)
0810 18:10:59.086375 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:59.086390 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:59.086429 (+ 39us) server_negotiation.cc:407] Connection header received
0810 18:10:59.087173 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:59.087193 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:59.087565 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:59.088027 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:59.088980 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:59.088985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:59.088996 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:59.089007 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:59.089072 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:59.089093 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:59.089185 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:59.089352 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 18:10:59.089530 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":160,"threads_started":1}
I20260810 18:10:59.093309 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:10:59.094314 14711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2321594e4ca74308a0bc1b563d177a29" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 45141 } }
I20260810 18:10:59.094627 14711 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:10:59.094729 14711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2321594e4ca74308a0bc1b563d177a29, State: Initialized, Role: FOLLOWER
I20260810 18:10:59.095274 14711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2321594e4ca74308a0bc1b563d177a29" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 45141 } }
I20260810 18:10:59.095461 14711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:10:59.095554 14711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:10:59.095701 14711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:10:59.097245 14711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2321594e4ca74308a0bc1b563d177a29" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 45141 } }
I20260810 18:10:59.097723 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:10:59.095781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46383 (local address 127.21.182.126:45141)
0810 18:10:59.095932 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:10:59.095936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:10:59.096162 (+ 226us) server_negotiation.cc:407] Connection header received
0810 18:10:59.096303 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:10:59.096305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:10:59.096336 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:10:59.096401 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:10:59.097172 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:10:59.097175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:10:59.097177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:10:59.097188 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:10:59.097214 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:10:59.097224 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:10:59.097309 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:10:59.097513 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 18:10:59.097550 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:10:59.097707 14711 leader_election.cc:302] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2321594e4ca74308a0bc1b563d177a29; no voters:
I20260810 18:10:59.098201 14711 leader_election.cc:288] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:10:59.098282 14722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:10:59.098515 14722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [term 1 LEADER]: Becoming Leader. State: Replica: 2321594e4ca74308a0bc1b563d177a29, State: Running, Role: LEADER
I20260810 18:10:59.098932 14722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2321594e4ca74308a0bc1b563d177a29" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 45141 } }
I20260810 18:10:59.099814 14711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:10:59.100894 14724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2321594e4ca74308a0bc1b563d177a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2321594e4ca74308a0bc1b563d177a29" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 45141 } } }
I20260810 18:10:59.101025 14724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:59.101380 14723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2321594e4ca74308a0bc1b563d177a29. Latest consensus state: current_term: 1 leader_uuid: "2321594e4ca74308a0bc1b563d177a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2321594e4ca74308a0bc1b563d177a29" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 45141 } } }
I20260810 18:10:59.101476 14740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:10:59.101539 14723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29 [sys.catalog]: This master's current role is: LEADER
I20260810 18:10:59.105091 14740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:10:59.111521 14740 catalog_manager.cc:1347] Generated new cluster ID: 9ab0ed90c372496dbefaed64cc1b1034
I20260810 18:10:59.111615 14740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:10:59.124337 14740 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:10:59.124816 14740 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:10:59.131321 14740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2321594e4ca74308a0bc1b563d177a29: Generated new TSK 0
I20260810 18:10:59.132187 14740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:10:59.137789 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 14634
2026-08-10T18:10:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1583 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:10:59 dist-test-slave-27sr krb5kdc[14754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:10:59 dist-test-slave-27sr krb5kdc[14754](info): set up 2 sockets
Aug 10 18:10:59 dist-test-slave-27sr krb5kdc[14754](info): commencing operation
krb5kdc: starting...
W20260810 18:11:00.268656 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:11:00 dist-test-slave-27sr krb5kdc[14754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:11:00.420156 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:33193
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:33193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:00.542466 14768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:00.542825 14768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:00.542914 14768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:00.547317 14768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:00.547432 14768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:00.547472 14768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:00.547533 14768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:00.547570 14768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:00.553313 14768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:33193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:33193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:00.554677 14768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:00.555815 14768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:00.565330 14777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:00.565339 14774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:00.565553 14768 server_base.cc:1034] running on GCE node
W20260810 18:11:00.565366 14779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:00.566411 14768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:00.567802 14768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:00.569111 14768 hybrid_clock.cc:648] HybridClock initialized: now 1786385460569078 us; error 46 us; skew 500 ppm
Aug 10 18:11:00 dist-test-slave-27sr krb5kdc[14754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385460, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:00.572468 14768 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:11:00.573557 14768 webserver.cc:460] Webserver started at http://127.21.182.126:42835/ using document root <none> and password file <none>
I20260810 18:11:00.574215 14768 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:00.574316 14768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:00.574580 14768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:00.576941 14768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "a1012aad75834d5f8f8d248fe61966fc"
format_stamp: "Formatted at 2026-08-10 18:11:00 on dist-test-slave-27sr"
server_key: "45dd60edfca9f9039c9ac1a2dca2ee81"
server_key_iv: "912264b95864b3db67f44b0fd83ce000"
server_key_version: "encryptionkey@0"
I20260810 18:11:00.577576 14768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "a1012aad75834d5f8f8d248fe61966fc"
format_stamp: "Formatted at 2026-08-10 18:11:00 on dist-test-slave-27sr"
server_key: "45dd60edfca9f9039c9ac1a2dca2ee81"
server_key_iv: "912264b95864b3db67f44b0fd83ce000"
server_key_version: "encryptionkey@0"
I20260810 18:11:00.581861 14768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:11:00.584656 14788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:00.586052 14768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:11:00.586221 14768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "a1012aad75834d5f8f8d248fe61966fc"
format_stamp: "Formatted at 2026-08-10 18:11:00 on dist-test-slave-27sr"
server_key: "45dd60edfca9f9039c9ac1a2dca2ee81"
server_key_iv: "912264b95864b3db67f44b0fd83ce000"
server_key_version: "encryptionkey@0"
I20260810 18:11:00.586350 14768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:00.615403 14768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:00.619318 14768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:00.619547 14768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:00.627172 14768 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:33193
I20260810 18:11:00.627200 14847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:33193 every 8 connection(s)
I20260810 18:11:00.628340 14768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:00.629168 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 14768
I20260810 18:11:00.629313 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:11:00.629604 22233 external_mini_cluster.cc:1428] Setting key 6ff74ac7d683d329b6b0eb88f688c4ab
I20260810 18:11:00.632032 14848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:00.637826 14848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc: Bootstrap starting.
Aug 10 18:11:00 dist-test-slave-27sr krb5kdc[14754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:11:00.640666 14848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:00.641438 14848 log.cc:824] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:00.643915 14848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc: No bootstrap required, opened a new log
I20260810 18:11:00.645234 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:00.631205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48397 (local address 127.21.182.126:33193)
0810 18:11:00.631693 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:00.631710 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:00.631751 (+ 41us) server_negotiation.cc:407] Connection header received
0810 18:11:00.632556 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:00.632576 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:00.632938 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:00.633326 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:00.634169 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:00.635207 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:00.636206 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:00.636688 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:00.636867 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:00.639992 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:00.639998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:00.640012 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:00.640044 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:00.642077 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:00.642642 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:00.642647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:00.642652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:00.642777 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:00.643076 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:00.643078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:00.643079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:00.643539 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:00.643722 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:00.644129 (+ 407us) server_negotiation.cc:299] Negotiation successful
0810 18:11:00.644483 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":143,"threads_started":1}
I20260810 18:11:00.648290 14848 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1012aad75834d5f8f8d248fe61966fc" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33193 } }
I20260810 18:11:00.648624 14848 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:00.648708 14848 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1012aad75834d5f8f8d248fe61966fc, State: Initialized, Role: FOLLOWER
I20260810 18:11:00.648698 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:11:00.649175 14848 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1012aad75834d5f8f8d248fe61966fc" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33193 } }
I20260810 18:11:00.649307 14848 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:00.649394 14848 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:00.649497 14848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:00.650632 14848 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1012aad75834d5f8f8d248fe61966fc" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33193 } }
I20260810 18:11:00.651057 14848 leader_election.cc:302] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1012aad75834d5f8f8d248fe61966fc; no voters:
I20260810 18:11:00.651511 14848 leader_election.cc:288] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:00.651583 14858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:00.651803 14858 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [term 1 LEADER]: Becoming Leader. State: Replica: a1012aad75834d5f8f8d248fe61966fc, State: Running, Role: LEADER
I20260810 18:11:00.652204 14858 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1012aad75834d5f8f8d248fe61966fc" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33193 } }
I20260810 18:11:00.652977 14848 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:00.653545 14858 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1012aad75834d5f8f8d248fe61966fc. Latest consensus state: current_term: 1 leader_uuid: "a1012aad75834d5f8f8d248fe61966fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1012aad75834d5f8f8d248fe61966fc" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33193 } } }
I20260810 18:11:00.654356 14858 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:00.654217 14859 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1012aad75834d5f8f8d248fe61966fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1012aad75834d5f8f8d248fe61966fc" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 33193 } } }
I20260810 18:11:00.654635 14859 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:00.655712 14867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:00.658965 14867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:00.662061 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:00.651123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37147 (local address 127.21.182.126:33193)
0810 18:11:00.651290 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:00.651295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:00.651454 (+ 159us) server_negotiation.cc:407] Connection header received
0810 18:11:00.651704 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:00.651709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:00.651775 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:00.651845 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:00.652917 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:00.653847 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:00.655146 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:00.655540 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:00.655674 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:00.657724 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:00.657728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:00.657733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:00.657771 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:00.659783 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:00.660411 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:00.660415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:00.660417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:00.660519 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:00.660909 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:00.660913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:00.660916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:00.661188 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:00.661339 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:00.661691 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 18:11:00.661853 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 18:11:00.665720 14867 catalog_manager.cc:1347] Generated new cluster ID: 01fb5cc9a6f44ffdb9d36e44910fbc05
I20260810 18:11:00.665818 14867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:00.694240 14867 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:00.695336 14867 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:00.700417 14867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1012aad75834d5f8f8d248fe61966fc: Generated new TSK 0
I20260810 18:11:00.701210 14867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:11:00.708703 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 14768
2026-08-10T18:11:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1570 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:00 dist-test-slave-27sr krb5kdc[14879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:00 dist-test-slave-27sr krb5kdc[14879](info): set up 2 sockets
Aug 10 18:11:00 dist-test-slave-27sr krb5kdc[14879](info): commencing operation
krb5kdc: starting...
W20260810 18:11:01.854954 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:11:01 dist-test-slave-27sr krb5kdc[14879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:11:02.012588 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:34735
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:40467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:34735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:02.135177 14894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:02.135501 14894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:02.135545 14894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:02.139932 14894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:02.140013 14894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:02.140038 14894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:02.140089 14894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:02.140128 14894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:02.146559 14894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:40467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:34735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:34735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:02.147935 14894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:02.149160 14894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:02.159509 14900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:02.159457 14901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:02.159457 14906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:02.159857 14894 server_base.cc:1034] running on GCE node
I20260810 18:11:02.160652 14894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:02.162070 14894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:02.163300 14894 hybrid_clock.cc:648] HybridClock initialized: now 1786385462163280 us; error 39 us; skew 500 ppm
Aug 10 18:11:02 dist-test-slave-27sr krb5kdc[14879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385462, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:02.166783 14894 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:11:02.168036 14894 webserver.cc:460] Webserver started at http://127.21.182.126:39151/ using document root <none> and password file <none>
I20260810 18:11:02.168697 14894 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:02.168772 14894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:02.168980 14894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:02.171450 14894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "1cae80b3588043029ac251b386f6ca36"
format_stamp: "Formatted at 2026-08-10 18:11:02 on dist-test-slave-27sr"
server_key: "cf37ffd40d7cb4f3e05ac82f41df4718"
server_key_iv: "ad3353488c5442eb5bf7cee707ad6616"
server_key_version: "encryptionkey@0"
I20260810 18:11:02.172184 14894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "1cae80b3588043029ac251b386f6ca36"
format_stamp: "Formatted at 2026-08-10 18:11:02 on dist-test-slave-27sr"
server_key: "cf37ffd40d7cb4f3e05ac82f41df4718"
server_key_iv: "ad3353488c5442eb5bf7cee707ad6616"
server_key_version: "encryptionkey@0"
I20260810 18:11:02.176718 14894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:11:02.179726 14913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:02.181308 14894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:11:02.181514 14894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "1cae80b3588043029ac251b386f6ca36"
format_stamp: "Formatted at 2026-08-10 18:11:02 on dist-test-slave-27sr"
server_key: "cf37ffd40d7cb4f3e05ac82f41df4718"
server_key_iv: "ad3353488c5442eb5bf7cee707ad6616"
server_key_version: "encryptionkey@0"
I20260810 18:11:02.181648 14894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:02.195597 14894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:02.198966 14894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:02.199198 14894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:02.206835 14894 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:34735
I20260810 18:11:02.206866 14977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:34735 every 8 connection(s)
I20260810 18:11:02.207935 14894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:02.210907 14978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:02.211297 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 14894
I20260810 18:11:02.211512 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:11:02.211850 22233 external_mini_cluster.cc:1428] Setting key e51dd5fe27569ed9ca70e2056bf56d32
I20260810 18:11:02.218211 14978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36: Bootstrap starting.
I20260810 18:11:02.220887 14978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:02.221787 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:02.213488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53995 (local address 127.21.182.126:34735)
0810 18:11:02.214036 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:02.214052 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:02.214090 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:11:02.214778 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:02.214798 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:02.215216 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:02.215597 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:02.216080 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:02.217100 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:02.218069 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:02.218625 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:02.218823 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:02.219762 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:02.219768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:02.219782 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:02.220294 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:02.220358 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:02.220386 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:02.220507 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:02.220636 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 18:11:02.220885 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":157,"threads_started":1}
I20260810 18:11:02.221827 14978 log.cc:824] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:02.224108 14978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36: No bootstrap required, opened a new log
I20260810 18:11:02.225003 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:11:02.228354 14978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cae80b3588043029ac251b386f6ca36" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34735 } }
I20260810 18:11:02.228602 14978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:02.228674 14978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cae80b3588043029ac251b386f6ca36, State: Initialized, Role: FOLLOWER
I20260810 18:11:02.229137 14978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cae80b3588043029ac251b386f6ca36" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34735 } }
I20260810 18:11:02.229276 14978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:02.229382 14978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:02.229502 14978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:02.230540 14978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cae80b3588043029ac251b386f6ca36" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34735 } }
I20260810 18:11:02.230938 14978 leader_election.cc:302] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cae80b3588043029ac251b386f6ca36; no voters:
I20260810 18:11:02.231294 14978 leader_election.cc:288] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:02.231751 14990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:02.232110 14990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [term 1 LEADER]: Becoming Leader. State: Replica: 1cae80b3588043029ac251b386f6ca36, State: Running, Role: LEADER
I20260810 18:11:02.232649 14983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:02.227264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49731 (local address 127.21.182.126:34735)
0810 18:11:02.227428 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:02.227433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:02.227571 (+ 138us) server_negotiation.cc:407] Connection header received
0810 18:11:02.227760 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:02.227764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:02.227820 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:02.227969 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:02.228280 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:02.229040 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:02.230161 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:02.230439 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:02.230570 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:02.231605 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:02.231610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:02.231613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:02.231631 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:02.231668 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:02.231683 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:02.232387 (+ 704us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:02.232427 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 18:11:02.232495 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:11:02.232578 14990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cae80b3588043029ac251b386f6ca36" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34735 } }
I20260810 18:11:02.232914 14978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:02.234139 14990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cae80b3588043029ac251b386f6ca36. Latest consensus state: current_term: 1 leader_uuid: "1cae80b3588043029ac251b386f6ca36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cae80b3588043029ac251b386f6ca36" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34735 } } }
I20260810 18:11:02.234302 14990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:02.234609 14991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cae80b3588043029ac251b386f6ca36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cae80b3588043029ac251b386f6ca36" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 34735 } } }
I20260810 18:11:02.234737 14991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:02.234807 14999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:02.239495 14999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:02.246398 14999 catalog_manager.cc:1347] Generated new cluster ID: eefe079715554d60af275e70ac406406
I20260810 18:11:02.246517 14999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:02.260929 14999 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:02.262355 14999 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:02.272691 14999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cae80b3588043029ac251b386f6ca36: Generated new TSK 0
I20260810 18:11:02.273742 14999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:11:02.287848 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 14894
2026-08-10T18:11:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1579 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:02 dist-test-slave-27sr krb5kdc[15014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:02 dist-test-slave-27sr krb5kdc[15014](info): set up 2 sockets
Aug 10 18:11:02 dist-test-slave-27sr krb5kdc[15014](info): commencing operation
krb5kdc: starting...
W20260810 18:11:03.452654 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:11:03 dist-test-slave-27sr krb5kdc[15014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:11:03.587673 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:45543
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:34897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:45543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:03.705039 15029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:03.705377 15029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:03.705462 15029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:03.709498 15029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:03.709594 15029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:03.709633 15029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:03.709694 15029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:03.709733 15029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:03.715200 15029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:34897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:45543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:45543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:03.716539 15029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:03.717610 15029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:03.726679 15035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:03.726882 15037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:03.726809 15029 server_base.cc:1034] running on GCE node
W20260810 18:11:03.726697 15034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:03.727566 15029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:03.728787 15029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:03.729981 15029 hybrid_clock.cc:648] HybridClock initialized: now 1786385463729967 us; error 26 us; skew 500 ppm
Aug 10 18:11:03 dist-test-slave-27sr krb5kdc[15014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385463, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:03.732988 15029 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:11:03.733955 15029 webserver.cc:460] Webserver started at http://127.21.182.126:34185/ using document root <none> and password file <none>
I20260810 18:11:03.734567 15029 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:03.734661 15029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:03.734908 15029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:03.737061 15029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "e500aa9fe8f3414b87c875adeab6ff19"
format_stamp: "Formatted at 2026-08-10 18:11:03 on dist-test-slave-27sr"
server_key: "a89e7d738c122778dc514d4867ea8332"
server_key_iv: "2f4aa9e7106d5c92f3bad2ad6121aacd"
server_key_version: "encryptionkey@0"
I20260810 18:11:03.737660 15029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "e500aa9fe8f3414b87c875adeab6ff19"
format_stamp: "Formatted at 2026-08-10 18:11:03 on dist-test-slave-27sr"
server_key: "a89e7d738c122778dc514d4867ea8332"
server_key_iv: "2f4aa9e7106d5c92f3bad2ad6121aacd"
server_key_version: "encryptionkey@0"
I20260810 18:11:03.741636 15029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 18:11:03.744235 15045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:03.745632 15029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:11:03.745743 15029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "e500aa9fe8f3414b87c875adeab6ff19"
format_stamp: "Formatted at 2026-08-10 18:11:03 on dist-test-slave-27sr"
server_key: "a89e7d738c122778dc514d4867ea8332"
server_key_iv: "2f4aa9e7106d5c92f3bad2ad6121aacd"
server_key_version: "encryptionkey@0"
I20260810 18:11:03.745854 15029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:03.750626 15029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:03.753636 15029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:03.753793 15029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:03.760944 15029 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:45543
I20260810 18:11:03.761005 15107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:45543 every 8 connection(s)
I20260810 18:11:03.761951 15029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:03.765038 15108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:03.765080 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 15029
I20260810 18:11:03.765229 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:11:03.765570 22233 external_mini_cluster.cc:1428] Setting key 82b45759a6380d52f67b67624dc0a918
I20260810 18:11:03.771688 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:03.766973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49063 (local address 127.21.182.126:45543)
0810 18:11:03.767444 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:03.767471 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:03.767516 (+ 45us) server_negotiation.cc:407] Connection header received
0810 18:11:03.768741 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:03.768768 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:03.769145 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:03.769532 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:03.770261 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:03.770267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:03.770282 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:03.770301 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:03.770364 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:03.770390 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:03.770534 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:03.770603 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 18:11:03.770758 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":132,"threads_started":1}
I20260810 18:11:03.772907 15108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19: Bootstrap starting.
I20260810 18:11:03.774627 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:11:03.775471 15108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:03.776309 15108 log.cc:824] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:03.778257 15108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19: No bootstrap required, opened a new log
I20260810 18:11:03.778504 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:03.776820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41691 (local address 127.21.182.126:45543)
0810 18:11:03.776947 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:03.776954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:03.777122 (+ 168us) server_negotiation.cc:407] Connection header received
0810 18:11:03.777246 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:03.777249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:03.777289 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:03.777375 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:03.778113 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:03.778117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:03.778119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:11:03.778131 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:03.778165 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:11:03.778177 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:03.778274 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:03.778345 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 18:11:03.778387 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:11:03.782090 15108 raft_consensus.cc:348] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e500aa9fe8f3414b87c875adeab6ff19" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 45543 } }
I20260810 18:11:03.782325 15108 raft_consensus.cc:374] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:11:03.782406 15108 raft_consensus.cc:729] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e500aa9fe8f3414b87c875adeab6ff19, State: Initialized, Role: FOLLOWER
I20260810 18:11:03.782886 15108 consensus_queue.cc:260] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e500aa9fe8f3414b87c875adeab6ff19" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 45543 } }
I20260810 18:11:03.783038 15108 raft_consensus.cc:388] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:11:03.783128 15108 raft_consensus.cc:482] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:11:03.783248 15108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:03.784274 15108 raft_consensus.cc:504] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e500aa9fe8f3414b87c875adeab6ff19" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 45543 } }
I20260810 18:11:03.784628 15108 leader_election.cc:302] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e500aa9fe8f3414b87c875adeab6ff19; no voters:
I20260810 18:11:03.784978 15108 leader_election.cc:288] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:11:03.785053 15123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:03.785264 15123 raft_consensus.cc:686] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [term 1 LEADER]: Becoming Leader. State: Replica: e500aa9fe8f3414b87c875adeab6ff19, State: Running, Role: LEADER
I20260810 18:11:03.785580 15123 consensus_queue.cc:237] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e500aa9fe8f3414b87c875adeab6ff19" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 45543 } }
I20260810 18:11:03.786386 15108 sys_catalog.cc:564] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:03.787444 15126 sys_catalog.cc:455] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e500aa9fe8f3414b87c875adeab6ff19. Latest consensus state: current_term: 1 leader_uuid: "e500aa9fe8f3414b87c875adeab6ff19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e500aa9fe8f3414b87c875adeab6ff19" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 45543 } } }
I20260810 18:11:03.787678 15124 sys_catalog.cc:455] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e500aa9fe8f3414b87c875adeab6ff19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e500aa9fe8f3414b87c875adeab6ff19" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 45543 } } }
I20260810 18:11:03.787789 15126 sys_catalog.cc:458] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:03.787869 15124 sys_catalog.cc:458] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:03.788371 15137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:03.791409 15137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:03.798344 15137 catalog_manager.cc:1347] Generated new cluster ID: 419160dcd6144a29ba9b97a3555a01ec
I20260810 18:11:03.798441 15137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:03.823858 15137 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:03.824324 15137 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:03.830483 15137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e500aa9fe8f3414b87c875adeab6ff19: Generated new TSK 0
I20260810 18:11:03.831343 15137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:11:03.842576 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 15029
2026-08-10T18:11:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1553 ms)
[----------] 18 tests from AuthTokenIssuingTest (28117 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:03 dist-test-slave-27sr krb5kdc[15149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:03 dist-test-slave-27sr krb5kdc[15149](info): set up 2 sockets
Aug 10 18:11:03 dist-test-slave-27sr krb5kdc[15149](info): commencing operation
krb5kdc: starting...
W20260810 18:11:04.979015 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:11:05.123555 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:38575
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:38575,127.21.182.125:44205,127.21.182.124:33035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:05.249454 15164 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:11:05.249802 15164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:05.249851 15164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:05.249922 15164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:05.253978 15164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:05.254073 15164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:05.254113 15164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:05.254169 15164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:05.254206 15164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:05.259678 15164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:38575,127.21.182.125:44205,127.21.182.124:33035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:38575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:05.260883 15164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:05.262002 15164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:05.270936 15171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:05.271045 15175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:05.270929 15172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:05.272128 15164 server_base.cc:1034] running on GCE node
I20260810 18:11:05.272722 15164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:05.273916 15164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:05.275084 15164 hybrid_clock.cc:648] HybridClock initialized: now 1786385465275072 us; error 36 us; skew 500 ppm
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:05.278308 15164 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:11:05.279359 15164 webserver.cc:460] Webserver started at http://127.21.182.126:38447/ using document root <none> and password file <none>
I20260810 18:11:05.280069 15164 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:05.280172 15164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:05.280429 15164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:05.282718 15164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "ef2d0dcd928c414ebeb71936be405f78"
format_stamp: "Formatted at 2026-08-10 18:11:05 on dist-test-slave-27sr"
server_key: "834efe2291558f8de032f647830d9ac4"
server_key_iv: "32422e1f4434e5aac6c324c3661c6af7"
server_key_version: "encryptionkey@0"
I20260810 18:11:05.283383 15164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "ef2d0dcd928c414ebeb71936be405f78"
format_stamp: "Formatted at 2026-08-10 18:11:05 on dist-test-slave-27sr"
server_key: "834efe2291558f8de032f647830d9ac4"
server_key_iv: "32422e1f4434e5aac6c324c3661c6af7"
server_key_version: "encryptionkey@0"
I20260810 18:11:05.288084 15164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 18:11:05.291275 15182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:05.293040 15164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:11:05.293241 15164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "ef2d0dcd928c414ebeb71936be405f78"
format_stamp: "Formatted at 2026-08-10 18:11:05 on dist-test-slave-27sr"
server_key: "834efe2291558f8de032f647830d9ac4"
server_key_iv: "32422e1f4434e5aac6c324c3661c6af7"
server_key_version: "encryptionkey@0"
I20260810 18:11:05.293398 15164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:05.307835 15164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:05.311045 15164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:05.311282 15164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:05.318924 15164 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:38575
I20260810 18:11:05.319231 15240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:38575 every 8 connection(s)
I20260810 18:11:05.320216 15164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:05.322055 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 15164
I20260810 18:11:05.322235 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:11:05.322558 22233 external_mini_cluster.cc:1428] Setting key a964d408bb7fa5a7ca18dc6da927b0ee
I20260810 18:11:05.324052 15241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:05.328486 15241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.182.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.125@KRBTEST.COM" created.
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:11:05.347841 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.331104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:38575 (local address 127.0.0.1:43866)
0810 18:11:05.331775 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.331813 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:05.332813 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:05.333598 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:05.333603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:05.334049 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:05.334349 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.334375 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.336137 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.336143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.337479 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.337497 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.338499 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.338503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.338786 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.339556 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:05.339577 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:05.341467 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:05.344225 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.344235 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.344246 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.344544 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.344987 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.344990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.344991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.345125 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.346077 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:05.346084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:05.346266 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 18:11:05.346459 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 18:11:05.346865 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":158,"thread_start_us":163,"threads_started":1}
I20260810 18:11:05.347852 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.331104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.21.182.126:38575)
0810 18:11:05.331601 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.331623 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.331982 (+ 359us) server_negotiation.cc:407] Connection header received
0810 18:11:05.333168 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.333173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.333384 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.333550 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.334563 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.335980 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.337630 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.338289 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.338493 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.341668 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.341679 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.341701 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.341739 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.344063 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.344676 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.344684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.344686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.344863 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.345248 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.345254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.345256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.345931 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.346099 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.346546 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.346778 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":330,"thread_start_us":170,"threads_started":1}
I20260810 18:11:05.350780 15241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:11:05.351689 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.351355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:44205 (local address 127.0.0.1:43382)
0810 18:11:05.351494 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.351576 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 18:11:05.352023 15184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.182.125:44205: connect: Connection refused (error 111)
I20260810 18:11:05.353241 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.352982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:44205 (local address 127.0.0.1:43392)
0810 18:11:05.353098 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.353147 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 18:11:05.353451 15241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.125:44205: Network error: Client connection negotiation failed: client connection to 127.21.182.125:44205: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.125.keytab.
Entry for principal kudu/127.21.182.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.125.keytab.
I20260810 18:11:05.355662 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.125:44205
--webserver_interface=127.21.182.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.125.keytab
--principal=kudu/127.21.182.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:38575,127.21.182.125:44205,127.21.182.124:33035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:11:05.402774 15241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } attempt: 1
I20260810 18:11:05.403760 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.403429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:44205 (local address 127.0.0.1:43398)
0810 18:11:05.403602 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.403671 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 18:11:05.404670 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.404404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:44205 (local address 127.0.0.1:43402)
0810 18:11:05.404542 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.404597 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 18:11:05.404904 15241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.125:44205: Network error: Client connection negotiation failed: client connection to 127.21.182.125:44205: connect: Connection refused (error 111)
I20260810 18:11:05.473098 15241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } attempt: 2
I20260810 18:11:05.474164 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.473811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:44205 (local address 127.0.0.1:43412)
0810 18:11:05.474004 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.474078 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 18:11:05.475241 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.474946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:44205 (local address 127.0.0.1:43416)
0810 18:11:05.475096 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.475156 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 18:11:05.475432 15241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.125:44205: Network error: Client connection negotiation failed: client connection to 127.21.182.125:44205: connect: Connection refused (error 111)
W20260810 18:11:05.481170 15252 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:11:05.481520 15252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:05.481546 15252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:05.481624 15252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:05.486043 15252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:05.486122 15252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:05.486143 15252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:05.486160 15252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:05.486176 15252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:05.493108 15252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:38575,127.21.182.125:44205,127.21.182.124:33035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.125.keytab
--principal=kudu/127.21.182.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.125:44205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.182.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:05.494518 15252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:05.495786 15252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:05.506335 15262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:05.506368 15258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:05.506356 15252 server_base.cc:1034] running on GCE node
W20260810 18:11:05.506417 15259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:05.507475 15252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:05.508880 15252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:05.510102 15252 hybrid_clock.cc:648] HybridClock initialized: now 1786385465510051 us; error 69 us; skew 500 ppm
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:05.513473 15252 init.cc:377] Logged in from keytab as kudu/127.21.182.125@KRBTEST.COM (short username kudu)
I20260810 18:11:05.514483 15252 webserver.cc:460] Webserver started at http://127.21.182.125:46579/ using document root <none> and password file <none>
I20260810 18:11:05.515079 15252 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:05.515144 15252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:05.515337 15252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:05.517707 15252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/data/instance:
uuid: "e314732a40a44958beff80fcdeef0daf"
format_stamp: "Formatted at 2026-08-10 18:11:05 on dist-test-slave-27sr"
server_key: "52631e5a07d3e0cf564164c8b5c6e788"
server_key_iv: "018640805efff843eef4b7ed000fce54"
server_key_version: "encryptionkey@0"
I20260810 18:11:05.518304 15252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/wal/instance:
uuid: "e314732a40a44958beff80fcdeef0daf"
format_stamp: "Formatted at 2026-08-10 18:11:05 on dist-test-slave-27sr"
server_key: "52631e5a07d3e0cf564164c8b5c6e788"
server_key_iv: "018640805efff843eef4b7ed000fce54"
server_key_version: "encryptionkey@0"
I20260810 18:11:05.522590 15252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 18:11:05.525341 15269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:05.526785 15252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 18:11:05.526942 15252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/wal
uuid: "e314732a40a44958beff80fcdeef0daf"
format_stamp: "Formatted at 2026-08-10 18:11:05 on dist-test-slave-27sr"
server_key: "52631e5a07d3e0cf564164c8b5c6e788"
server_key_iv: "018640805efff843eef4b7ed000fce54"
server_key_version: "encryptionkey@0"
I20260810 18:11:05.527053 15252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:05.559181 15252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:05.563174 15252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:05.563426 15252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:05.566625 15241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } attempt: 3
I20260810 18:11:05.567642 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.567204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:44205 (local address 127.0.0.1:43422)
0810 18:11:05.567425 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.567517 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 18:11:05.568641 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.568331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:44205 (local address 127.0.0.1:43426)
0810 18:11:05.568474 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.568558 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 18:11:05.568811 15241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.125:44205: Network error: Client connection negotiation failed: client connection to 127.21.182.125:44205: connect: Connection refused (error 111)
I20260810 18:11:05.571841 15252 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.125:44205
I20260810 18:11:05.572146 15330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.125:44205 every 8 connection(s)
I20260810 18:11:05.573029 15252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/data/info.pb
I20260810 18:11:05.574448 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 15252
I20260810 18:11:05.574595 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-1/wal/instance
I20260810 18:11:05.574916 22233 external_mini_cluster.cc:1428] Setting key 784934702df9cae57c6b4ee29feccda2
I20260810 18:11:05.576192 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:05.580107 15331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.182.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.124@KRBTEST.COM" created.
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.125@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:11:05.596347 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.582162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.21.182.126:38575)
0810 18:11:05.582321 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.582325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.583502 (+ 1177us) server_negotiation.cc:407] Connection header received
0810 18:11:05.584618 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.584622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.584703 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.584811 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.586113 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.586736 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.587960 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.588211 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.588329 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.591489 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.591492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.591494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.591522 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.593243 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.593990 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.593994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.593995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.594079 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.594594 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.594598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.594599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.594899 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.594994 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.596064 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.596193 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:11:05.597397 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.582704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:38575 (local address 127.0.0.1:43870)
0810 18:11:05.583318 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.583363 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:05.584368 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:05.584995 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:05.585002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:05.585416 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:05.585961 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.585975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.586880 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.586884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.587809 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.587819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.588376 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.588380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.588555 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.589280 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:05.589306 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:05.591236 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:05.593391 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.593402 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.593422 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.593850 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.594180 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.594187 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.594191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.594453 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.595037 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:05.595046 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:05.595447 (+ 401us) client_negotiation.cc:769] Sending connection context
0810 18:11:05.595715 (+ 268us) client_negotiation.cc:240] Negotiation successful
0810 18:11:05.596166 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":173,"threads_started":1}
I20260810 18:11:05.599010 15331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.125@KRBTEST.COM for kudu/127.21.182.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.124.keytab.
Entry for principal kudu/127.21.182.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.124.keytab.
I20260810 18:11:05.607384 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.124:33035
--webserver_interface=127.21.182.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.124.keytab
--principal=kudu/127.21.182.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:38575,127.21.182.125:44205,127.21.182.124:33035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:11:05.609859 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.599645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:44205 (local address 127.0.0.1:43428)
0810 18:11:05.599877 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.599915 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:05.600015 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:05.600767 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:05.600770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:05.600980 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:05.601068 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.601074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.602307 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.602315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.602950 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.602959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.603685 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.603689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.603783 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.604350 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:05.604386 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:05.605733 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:05.607881 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.607888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.607912 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.608221 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.608593 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.608600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.608605 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.608734 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.609366 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:05.609372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:05.609474 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 18:11:05.609574 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 18:11:05.609716 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260810 18:11:05.610421 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.599944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.21.182.125:44205)
0810 18:11:05.600346 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.600357 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.600378 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:11:05.600416 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.600420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.600596 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.600772 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.601191 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.602188 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.603078 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.603557 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.603686 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.605870 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.605879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.605892 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.605914 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.607766 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.608382 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.608391 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.608395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.608491 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.608844 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.608849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.608854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.609173 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.609323 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.610182 (+ 859us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.610308 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":185,"threads_started":1}
I20260810 18:11:05.612573 15331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:11:05.613481 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.613221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33035 (local address 127.0.0.1:36570)
0810 18:11:05.613346 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.613422 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 18:11:05.613685 15270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
I20260810 18:11:05.614635 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.614405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33035 (local address 127.0.0.1:36586)
0810 18:11:05.614527 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.614583 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 18:11:05.614820 15331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:33035: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
I20260810 18:11:05.664052 15331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } attempt: 1
I20260810 18:11:05.665038 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.664705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33035 (local address 127.0.0.1:36596)
0810 18:11:05.664889 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.664955 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 18:11:05.665809 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.665578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33035 (local address 127.0.0.1:36604)
0810 18:11:05.665691 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.665746 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 18:11:05.665948 15331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:33035: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
I20260810 18:11:05.711966 15241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } attempt: 4
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for kudu/127.21.182.125@KRBTEST.COM
I20260810 18:11:05.722688 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.712693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.21.182.125:44205)
0810 18:11:05.712841 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.712845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.712901 (+ 56us) server_negotiation.cc:407] Connection header received
0810 18:11:05.713067 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.713069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.713118 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.713183 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.713756 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.714667 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.716463 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.716707 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.716800 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.719069 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.719073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.719078 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.719132 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.720945 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.721393 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.721397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.721399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.721491 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.721809 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.721812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.721814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.722010 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.722118 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.722416 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.722536 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:11:05.722688 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.712553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:44205 (local address 127.0.0.1:43436)
0810 18:11:05.712808 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.712827 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:05.712948 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:05.713245 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:05.713248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:05.713488 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:05.713622 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.713632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.714843 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.714848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.716339 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.716354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.716829 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.716833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.716945 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.717465 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:05.717481 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:05.718916 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:05.721065 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.721068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.721071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.721288 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.721592 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.721595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.721597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.721692 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.722124 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:05.722129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:05.722235 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 18:11:05.722347 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 18:11:05.722501 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260810 18:11:05.723538 15241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:11:05.724309 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.724078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33035 (local address 127.0.0.1:36612)
0810 18:11:05.724196 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.724245 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 18:11:05.724972 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.724737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33035 (local address 127.0.0.1:36622)
0810 18:11:05.724874 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.724927 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 18:11:05.725121 15241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:33035: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
W20260810 18:11:05.728785 15338 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:11:05.729243 15338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:05.729321 15338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:05.729394 15338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:05.733521 15338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:05.733606 15338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:05.733654 15338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:05.733690 15338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:05.733723 15338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:05.734112 15331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } attempt: 2
I20260810 18:11:05.734860 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.734590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33035 (local address 127.0.0.1:36624)
0810 18:11:05.734719 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.734787 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 18:11:05.735615 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.735394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33035 (local address 127.0.0.1:36636)
0810 18:11:05.735510 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.735560 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 18:11:05.735745 15331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:33035: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
I20260810 18:11:05.739584 15338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:38575,127.21.182.125:44205,127.21.182.124:33035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.124.keytab
--principal=kudu/127.21.182.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.124:33035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.182.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:05.740803 15338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:05.741885 15338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:05.751287 15351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:05.751336 15347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:05.751281 15346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:05.751339 15338 server_base.cc:1034] running on GCE node
I20260810 18:11:05.752173 15338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:05.753490 15338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:05.754663 15338 hybrid_clock.cc:648] HybridClock initialized: now 1786385465754640 us; error 47 us; skew 500 ppm
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:05.757663 15338 init.cc:377] Logged in from keytab as kudu/127.21.182.124@KRBTEST.COM (short username kudu)
I20260810 18:11:05.758538 15338 webserver.cc:460] Webserver started at http://127.21.182.124:44113/ using document root <none> and password file <none>
I20260810 18:11:05.759130 15338 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:05.759227 15338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:05.759524 15338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:05.761688 15338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/data/instance:
uuid: "e3ce4409c4374d7c8989a9bdcf00445f"
format_stamp: "Formatted at 2026-08-10 18:11:05 on dist-test-slave-27sr"
server_key: "cd384437ac0100ee3c54971460f85d14"
server_key_iv: "24608235ba44b6a80a6f23e57a30be5b"
server_key_version: "encryptionkey@0"
I20260810 18:11:05.762212 15338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/wal/instance:
uuid: "e3ce4409c4374d7c8989a9bdcf00445f"
format_stamp: "Formatted at 2026-08-10 18:11:05 on dist-test-slave-27sr"
server_key: "cd384437ac0100ee3c54971460f85d14"
server_key_iv: "24608235ba44b6a80a6f23e57a30be5b"
server_key_version: "encryptionkey@0"
I20260810 18:11:05.766121 15338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:11:05.768690 15359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:05.769958 15338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:11:05.770061 15338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/wal
uuid: "e3ce4409c4374d7c8989a9bdcf00445f"
format_stamp: "Formatted at 2026-08-10 18:11:05 on dist-test-slave-27sr"
server_key: "cd384437ac0100ee3c54971460f85d14"
server_key_iv: "24608235ba44b6a80a6f23e57a30be5b"
server_key_version: "encryptionkey@0"
I20260810 18:11:05.770148 15338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:05.784281 15241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } attempt: 1
I20260810 18:11:05.785209 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.784883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33035 (local address 127.0.0.1:36638)
0810 18:11:05.785051 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.785124 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 18:11:05.786329 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.785958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33035 (local address 127.0.0.1:36654)
0810 18:11:05.786170 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.786235 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 18:11:05.786514 15241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:33035: Network error: Client connection negotiation failed: client connection to 127.21.182.124:33035: connect: Connection refused (error 111)
I20260810 18:11:05.788868 15338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:05.792412 15338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:05.792622 15338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:05.800469 15338 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.124:33035
I20260810 18:11:05.800901 15421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.124:33035 every 8 connection(s)
I20260810 18:11:05.801541 15338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/data/info.pb
I20260810 18:11:05.804723 15423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:05.806210 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 15338
I20260810 18:11:05.806368 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786385164390354-22233-0/minicluster-data/master-2/wal/instance
I20260810 18:11:05.806730 22233 external_mini_cluster.cc:1428] Setting key e7126e1d862b2ac4167ebd3e4ad2773e
I20260810 18:11:05.809818 15423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.124@KRBTEST.COM
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.124@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:11:05.821305 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.811668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.21.182.126:38575)
0810 18:11:05.811815 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.811819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.812089 (+ 270us) server_negotiation.cc:407] Connection header received
0810 18:11:05.812233 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.812236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.812284 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.812366 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.813024 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.813547 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.814428 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.814651 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.814759 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.817642 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.817645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.817647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.817672 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.819200 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.819756 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.819759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.819761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.819842 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.820184 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.820190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.820192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.820500 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.820596 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.821081 (+ 485us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.821183 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:11:05.821951 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.808393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.21.182.124:33035)
0810 18:11:05.808852 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.808867 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.808903 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:11:05.809540 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.809557 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.809923 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.810282 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.811168 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.812513 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.813476 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.813982 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.814150 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.816977 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.816984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.816998 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.817024 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.818665 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.819113 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.819119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.819125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.819238 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.819560 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.819563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.819565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.820120 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.820311 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.820601 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.820898 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":136,"threads_started":1}
I20260810 18:11:05.821951 15427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.811575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:38575 (local address 127.0.0.1:43880)
0810 18:11:05.811957 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.812007 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:05.812139 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:05.812422 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:05.812426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:05.812763 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:05.812895 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.812900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.813674 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.813678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.814296 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.814305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.814799 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.814803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.814914 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.815500 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:05.815515 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:05.817468 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:05.819309 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.819320 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.819329 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.819652 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.819974 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.819977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.819978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.820072 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.820598 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:05.820604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:05.820714 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 18:11:05.820967 (+ 253us) client_negotiation.cc:240] Negotiation successful
0810 18:11:05.821124 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"spinlock_wait_cycles":59008,"thread_start_us":97,"threads_started":1}
I20260810 18:11:05.824100 15423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:11:05.826942 15331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } attempt: 3
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.124@KRBTEST.COM for kudu/127.21.182.125@KRBTEST.COM
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.125@KRBTEST.COM for kudu/127.21.182.124@KRBTEST.COM
I20260810 18:11:05.834447 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.824762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.21.182.125:44205)
0810 18:11:05.824896 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.824901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.824916 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:11:05.824997 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.824999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.825042 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.825123 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.825637 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.826384 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.827491 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.827831 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.827996 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.830414 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.830418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.830420 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.830449 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.832601 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.833041 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.833045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.833047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.833135 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.833454 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.833460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.833462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.833712 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.833823 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.834145 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.834281 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:11:05.834447 15427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.824637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:44205 (local address 127.0.0.1:43446)
0810 18:11:05.824816 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.824830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:05.824910 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:05.825147 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:05.825149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:05.825359 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:05.825506 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.825515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.826498 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.826500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.827329 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.827340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.827993 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.828000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.828096 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.828600 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:05.828612 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:05.830277 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:05.832713 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.832717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.832719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.832934 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.833247 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.833252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.833256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.833360 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.833822 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:05.833825 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:05.833934 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 18:11:05.834061 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 18:11:05.834299 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260810 18:11:05.835170 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.824839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.21.182.126:38575)
0810 18:11:05.824943 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.824947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.824962 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:11:05.824996 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.824998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.825040 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.825123 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.825852 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.826549 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.827671 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.827935 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.828036 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.831079 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.831084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.831088 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.831119 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.833037 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.833514 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.833518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.833520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.833607 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.833920 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.833925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.833928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.834239 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.834503 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.834895 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.835018 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:11:05.835193 15423 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:11:05.837910 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.827389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33035 (local address 127.0.0.1:36674)
0810 18:11:05.827560 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.827573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:05.827647 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:05.828007 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:05.828009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:05.828178 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:05.828244 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.828248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.829182 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.829187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.830058 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.830070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.830657 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.830660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.830759 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.831264 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:05.831283 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:05.832764 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:05.835525 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.835530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.835534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.835773 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.836805 (+ 1032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.836809 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.836812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.836916 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.837420 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:05.837423 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:05.837515 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 18:11:05.837626 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 18:11:05.837753 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260810 18:11:05.837953 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.827412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36674 (local address 127.21.182.124:33035)
0810 18:11:05.827775 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.827780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.827791 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:11:05.827822 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.827825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.827867 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.828386 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.828483 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.829065 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.830186 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.830547 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.830695 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.832930 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.832935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.832939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.832967 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.835415 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.836595 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.836598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.836600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.836689 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.837002 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.837008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.837013 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.837271 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.837419 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.837685 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.837806 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297}
I20260810 18:11:05.841404 15331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf: Bootstrap starting.
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.124@KRBTEST.COM for kudu/127.21.182.124@KRBTEST.COM
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.125@KRBTEST.COM
I20260810 18:11:05.844549 15331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:05.845466 15331 log.cc:824] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:05.846232 15427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.835672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33035 (local address 127.0.0.1:36686)
0810 18:11:05.835799 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.835815 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:05.835882 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:05.837009 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:05.837011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:05.837157 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:05.837221 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.837226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.838621 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.838624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.839398 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.839408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.839997 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.840001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.840163 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.840710 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:05.840725 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:05.842225 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:05.844559 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.844564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.844566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.844880 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.845172 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.845175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.845177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.845250 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.845751 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:05.845754 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:05.845856 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 18:11:05.845986 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 18:11:05.846112 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":23}
I20260810 18:11:05.846570 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.836362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36686 (local address 127.21.182.124:33035)
0810 18:11:05.836719 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.836729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.836742 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:11:05.836778 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.836782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.836822 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.836902 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.837340 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.838502 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.839519 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.839853 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.840063 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.842364 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.842368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.842370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.842392 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.844435 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.844994 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.844997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.844998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.845083 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.845341 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.845346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.845348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.845637 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.845746 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.846056 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.846187 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":285,"thread_start_us":106,"threads_started":1}
I20260810 18:11:05.847632 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.836498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.21.182.125:44205)
0810 18:11:05.836609 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.836613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.836625 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:11:05.836704 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.836709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.836762 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.836839 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.837578 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.838265 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.839315 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.839622 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.839679 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.843028 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.843032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.843034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.843065 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.845022 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.845517 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.845520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.845522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.845654 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.846647 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.846653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.846658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.846858 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.847008 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.847389 (+ 381us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.847500 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":46}
I20260810 18:11:05.848158 15331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf: No bootstrap required, opened a new log
I20260810 18:11:05.849552 15423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f: Bootstrap starting.
I20260810 18:11:05.851833 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:11:05.852237 15331 raft_consensus.cc:348] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } }
I20260810 18:11:05.852404 15423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:05.852455 15331 raft_consensus.cc:729] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e314732a40a44958beff80fcdeef0daf, State: Initialized, Role: FOLLOWER
I20260810 18:11:05.852766 15331 consensus_queue.cc:260] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } }
I20260810 18:11:05.853384 15423 log.cc:824] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:05.853722 15241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } attempt: 2
I20260810 18:11:05.854241 15331 sys_catalog.cc:564] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:05.853390 15211 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ef2d0dcd928c414ebeb71936be405f78"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43894
I20260810 18:11:05.855190 15430 sys_catalog.cc:455] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } } }
I20260810 18:11:05.855378 15430 sys_catalog.cc:458] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:05.858057 15423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f: No bootstrap required, opened a new log
Aug 10 18:11:05 dist-test-slave-27sr krb5kdc[15149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385465, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for kudu/127.21.182.124@KRBTEST.COM
W20260810 18:11:05.861810 15446 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:11:05.862003 15446 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:11:05.857920 15252 master_runner.cc:186] Error getting master registration for 127.21.182.126:38575: OK, instance_id { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" instance_seqno: 1786385465316723 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:11:05.862237 15252 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:11:05.862879 15423 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } }
I20260810 18:11:05.863161 15423 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ce4409c4374d7c8989a9bdcf00445f, State: Initialized, Role: FOLLOWER
I20260810 18:11:05.863548 15423 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } }
I20260810 18:11:05.864212 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.854110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:33035 (local address 127.0.0.1:36696)
0810 18:11:05.854280 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:05.854296 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:05.854386 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:05.854846 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:05.854849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:05.855101 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:05.855225 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.855233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.856317 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.856323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.857140 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:05.857153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.857865 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.857868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:05.857948 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.858535 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:05.858551 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:05.860178 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:05.862312 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.862315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.862317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.862572 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.862913 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:05.862916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:05.862918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:05.863032 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:05.863530 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:05.863533 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:05.863665 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 18:11:05.863852 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 18:11:05.864030 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 18:11:05.864404 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.854149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.21.182.124:33035)
0810 18:11:05.854532 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.854536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.854548 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:11:05.854588 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.854591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.854632 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.854708 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.855375 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.856190 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.857304 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.857634 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.857736 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.860354 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.860358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.860361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.860389 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.862162 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.862691 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.862694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.862697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.862808 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.863154 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.863158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.863160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.863421 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.863523 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.863990 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.864189 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309}
I20260810 18:11:05.865088 15423 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:05.868024 15241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78: Bootstrap starting.
W20260810 18:11:05.870518 15464 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:11:05.870594 15464 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:11:05.870954 15449 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } } }
I20260810 18:11:05.871027 15241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:05.871069 15449 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:05.872292 15241 log.cc:824] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:05.874610 15241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78: No bootstrap required, opened a new log
I20260810 18:11:05.872916 15338 master_runner.cc:186] Error getting master registration for 127.21.182.126:38575: OK, instance_id { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" instance_seqno: 1786385465316723 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:11:05.876498 15338 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:11:05.877843 15241 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } }
I20260810 18:11:05.878091 15241 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2d0dcd928c414ebeb71936be405f78, State: Initialized, Role: FOLLOWER
I20260810 18:11:05.878479 15241 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } }
I20260810 18:11:05.879189 15467 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } } }
I20260810 18:11:05.879493 15467 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:05.880061 15241 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:05.880113 15211 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:11:05.880234 15211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:05.881546 15211 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } }
I20260810 18:11:05.883127 15390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef2d0dcd928c414ebeb71936be405f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e3ce4409c4374d7c8989a9bdcf00445f"
I20260810 18:11:05.883168 15301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef2d0dcd928c414ebeb71936be405f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e314732a40a44958beff80fcdeef0daf"
I20260810 18:11:05.883384 15390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:05.883420 15301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:05.884886 15390 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef2d0dcd928c414ebeb71936be405f78 in term 1.
I20260810 18:11:05.884949 15301 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef2d0dcd928c414ebeb71936be405f78 in term 1.
I20260810 18:11:05.885339 15183 leader_election.cc:302] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3ce4409c4374d7c8989a9bdcf00445f, ef2d0dcd928c414ebeb71936be405f78; no voters:
I20260810 18:11:05.885627 15467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:05.885890 15467 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [term 1 LEADER]: Becoming Leader. State: Replica: ef2d0dcd928c414ebeb71936be405f78, State: Running, Role: LEADER
I20260810 18:11:05.886266 15467 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } }
I20260810 18:11:05.882529 15211 leader_election.cc:288] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [CANDIDATE]: Term 1 election: Requested vote from peers e314732a40a44958beff80fcdeef0daf (127.21.182.125:44205), e3ce4409c4374d7c8989a9bdcf00445f (127.21.182.124:33035)
I20260810 18:11:05.888824 15481 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef2d0dcd928c414ebeb71936be405f78. Latest consensus state: current_term: 1 leader_uuid: "ef2d0dcd928c414ebeb71936be405f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } } }
I20260810 18:11:05.888953 15481 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:05.889518 15489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:05.889662 15164 master_runner.cc:263] No leader master found from master ef2d0dcd928c414ebeb71936be405f78
I20260810 18:11:05.889726 15164 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:11:05.893006 15489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:05.932260 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.895451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36710 (local address 127.21.182.124:33035)
0810 18:11:05.895594 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.895599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.897723 (+ 2124us) server_negotiation.cc:407] Connection header received
0810 18:11:05.899120 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.899124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.899187 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.899245 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.902437 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.903265 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.910633 (+ 7368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.911017 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.911116 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.922962 (+ 11846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.922966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.922969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.923005 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.924790 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.926230 (+ 1440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.926234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.926236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.926341 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.929718 (+ 3377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.929722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.929724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.930066 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.930205 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.931877 (+ 1672us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.932054 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:11:05.932963 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.895876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.21.182.126:38575)
0810 18:11:05.896005 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.896009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.897513 (+ 1504us) server_negotiation.cc:407] Connection header received
0810 18:11:05.898025 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.898028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.898079 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.898133 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.904476 (+ 6343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.905265 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.911974 (+ 6709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.912332 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.912412 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.920871 (+ 8459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.920876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.920879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.920920 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.922707 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.929255 (+ 6548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.929260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.929261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.929372 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.930558 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.930562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.930564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.930841 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.930978 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.932648 (+ 1670us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.932802 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:11:05.937335 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:05.896638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.21.182.125:44205)
0810 18:11:05.896761 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:05.896765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:05.899700 (+ 2935us) server_negotiation.cc:407] Connection header received
0810 18:11:05.899973 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:05.899978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:05.900037 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:05.900111 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:05.906503 (+ 6392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.907309 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.917179 (+ 9870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:05.917572 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:05.917663 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:05.926536 (+ 8873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:05.926540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:05.926544 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:05.926577 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:05.928511 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.931378 (+ 2867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.931382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.931384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.931485 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:05.933232 (+ 1747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:05.933237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:05.933239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:05.933484 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:05.933601 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:05.937003 (+ 3402us) server_negotiation.cc:299] Negotiation successful
0810 18:11:05.937160 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:11:05.940485 15390 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f [term 1 FOLLOWER]: Refusing update from remote peer ef2d0dcd928c414ebeb71936be405f78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:05.941327 15481 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:05.946359 15449 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef2d0dcd928c414ebeb71936be405f78. Latest consensus state: current_term: 1 leader_uuid: "ef2d0dcd928c414ebeb71936be405f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } } }
I20260810 18:11:05.946525 15449 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:05.950628 15301 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf [term 1 FOLLOWER]: Refusing update from remote peer ef2d0dcd928c414ebeb71936be405f78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:05.951355 15481 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:05.957317 15467 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef2d0dcd928c414ebeb71936be405f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } } }
I20260810 18:11:05.957486 15467 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:05.957983 15449 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef2d0dcd928c414ebeb71936be405f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } } }
I20260810 18:11:05.958127 15449 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:05.958267 15496 mvcc.cc:205] Tried to move back new op lower bound from 7317034868387852288 to 7317034868274053120. Current Snapshot: MvccSnapshot[applied={T|T < 7317034868387852288}]
I20260810 18:11:05.960615 15430 sys_catalog.cc:455] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef2d0dcd928c414ebeb71936be405f78. Latest consensus state: current_term: 1 leader_uuid: "ef2d0dcd928c414ebeb71936be405f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } } }
I20260810 18:11:05.960826 15430 sys_catalog.cc:458] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:05.961745 15491 mvcc.cc:205] Tried to move back new op lower bound from 7317034868387852288 to 7317034868274053120. Current Snapshot: MvccSnapshot[applied={T|T < 7317034868387852288}]
I20260810 18:11:05.972502 15481 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef2d0dcd928c414ebeb71936be405f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } } }
I20260810 18:11:05.972693 15481 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:05.974306 15430 sys_catalog.cc:455] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef2d0dcd928c414ebeb71936be405f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d0dcd928c414ebeb71936be405f78" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 38575 } } peers { permanent_uuid: "e314732a40a44958beff80fcdeef0daf" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 44205 } } peers { permanent_uuid: "e3ce4409c4374d7c8989a9bdcf00445f" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 33035 } } }
I20260810 18:11:05.974470 15430 sys_catalog.cc:458] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:05.978062 15489 catalog_manager.cc:1347] Generated new cluster ID: 9434052b90d74e22a6477156b4843962
I20260810 18:11:05.978158 15489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:05.989173 15502 mvcc.cc:205] Tried to move back new op lower bound from 7317034868387852288 to 7317034868274053120. Current Snapshot: MvccSnapshot[applied={T|T < 7317034868387852288}]
I20260810 18:11:06.015677 15489 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:06.017076 15489 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:06.033215 15489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef2d0dcd928c414ebeb71936be405f78: Generated new TSK 0
I20260810 18:11:06.034125 15489 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:11:06.078167 15245 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:06.074727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.21.182.126:38575)
0810 18:11:06.074870 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:06.074876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:06.076969 (+ 2093us) server_negotiation.cc:407] Connection header received
0810 18:11:06.077180 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:06.077184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:06.077238 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:06.077292 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:06.078015 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43900: BlockingRecv error: recv got EOF from 127.0.0.1:43900 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 18:11:06.080143 15337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:06.075543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.21.182.125:44205)
0810 18:11:06.075657 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:06.075662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:06.078720 (+ 3058us) server_negotiation.cc:407] Connection header received
0810 18:11:06.079176 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:06.079180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:06.079243 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:06.079311 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:06.079976 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43472: BlockingRecv error: recv got EOF from 127.0.0.1:43472 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 18:11:06.081383 15428 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:06.076321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36720 (local address 127.21.182.124:33035)
0810 18:11:06.076435 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:06.076439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:06.079078 (+ 2639us) server_negotiation.cc:407] Connection header received
0810 18:11:06.080533 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:06.080537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:06.080590 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:06.080643 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:06.081242 (+ 599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36720: BlockingRecv error: recv got EOF from 127.0.0.1:36720 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 18:11:06.086863 15337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:06.085044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.21.182.125:44205)
0810 18:11:06.085191 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:06.085197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:06.085733 (+ 536us) server_negotiation.cc:407] Connection header received
0810 18:11:06.085994 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:06.085997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:06.086083 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:06.086138 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:06.086728 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43486: BlockingRecv error: recv got EOF from 127.0.0.1:43486 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 18:11:06.093292 15337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:06.091172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.21.182.125:44205)
0810 18:11:06.091338 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:06.091344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:06.092152 (+ 808us) server_negotiation.cc:407] Connection header received
0810 18:11:06.092384 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:06.092388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:06.092449 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:06.092510 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:06.093162 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43490: BlockingRecv error: recv got EOF from 127.0.0.1:43490 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 18:11:06.099582 15337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:06.097786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.21.182.125:44205)
0810 18:11:06.097915 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:06.097919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:06.098489 (+ 570us) server_negotiation.cc:407] Connection header received
0810 18:11:06.098718 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:06.098721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:06.098773 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:06.098824 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:06.099433 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43502: BlockingRecv error: recv got EOF from 127.0.0.1:43502 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 18:11:06.107649 15337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:06.106298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.21.182.125:44205)
0810 18:11:06.106482 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:06.106488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:06.106696 (+ 208us) server_negotiation.cc:407] Connection header received
0810 18:11:06.106874 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:06.106878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:06.106937 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:06.107033 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:06.107520 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43506: BlockingRecv error: recv got EOF from 127.0.0.1:43506 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260810 18:11:06.119516 15337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:06.118109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.21.182.125:44205)
0810 18:11:06.118299 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:06.118304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:06.118463 (+ 159us) server_negotiation.cc:407] Connection header received
0810 18:11:06.118671 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:06.118675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:06.118739 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:06.118836 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:06.119371 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43512: BlockingRecv error: recv got EOF from 127.0.0.1:43512 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260810 18:11:06.139108 15337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:06.137974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.21.182.125:44205)
0810 18:11:06.138119 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:06.138123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:06.138233 (+ 110us) server_negotiation.cc:407] Connection header received
0810 18:11:06.138370 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:06.138373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:06.138426 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:06.138514 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:06.138999 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43518: BlockingRecv error: recv got EOF from 127.0.0.1:43518 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 18:11:06.174924 15337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:06.173612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43526 (local address 127.21.182.125:44205)
0810 18:11:06.173836 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:06.173841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:06.173962 (+ 121us) server_negotiation.cc:407] Connection header received
0810 18:11:06.174119 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:06.174125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:06.174186 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:06.174290 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:06.174745 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43526: BlockingRecv error: recv got EOF from 127.0.0.1:43526 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260810 18:11:06.242730 15337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:06.241378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43536 (local address 127.21.182.125:44205)
0810 18:11:06.241573 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:06.241578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:06.241730 (+ 152us) server_negotiation.cc:407] Connection header received
0810 18:11:06.241933 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:06.241937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:06.241999 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:06.242100 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:06.242612 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43536: BlockingRecv error: recv got EOF from 127.0.0.1:43536 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260810 18:11:06.374485 15337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:06.373130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.21.182.125:44205)
0810 18:11:06.373322 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:06.373327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:06.373485 (+ 158us) server_negotiation.cc:407] Connection header received
0810 18:11:06.373667 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:06.373670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:06.373728 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:06.373829 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:06.374339 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43540: BlockingRecv error: recv got EOF from 127.0.0.1:43540 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 18:11:06.634457 15337 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:06.633153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.21.182.125:44205)
0810 18:11:06.633348 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:06.633355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:06.633442 (+ 87us) server_negotiation.cc:407] Connection header received
0810 18:11:06.633636 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:06.633641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:06.633696 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:06.633793 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:06.634307 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43548: BlockingRecv error: recv got EOF from 127.0.0.1:43548 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 18:11:06.863829 15446 catalog_manager.cc:1259] Loaded cluster ID: 9434052b90d74e22a6477156b4843962
I20260810 18:11:06.863958 15446 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf: loading cluster ID for follower catalog manager: success
I20260810 18:11:06.865854 15446 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf: acquiring CA information for follower catalog manager: success
I20260810 18:11:06.867288 15446 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e314732a40a44958beff80fcdeef0daf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:11:06.871812 15464 catalog_manager.cc:1259] Loaded cluster ID: 9434052b90d74e22a6477156b4843962
I20260810 18:11:06.871953 15464 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f: loading cluster ID for follower catalog manager: success
I20260810 18:11:06.873960 15464 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f: acquiring CA information for follower catalog manager: success
I20260810 18:11:06.875312 15464 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e3ce4409c4374d7c8989a9bdcf00445f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:11:07.153131 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:07.148900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.21.182.125:44205)
0810 18:11:07.149198 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:07.149203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:07.149221 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:11:07.149371 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:07.149374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:07.149529 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:07.149682 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:07.149973 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:07.150534 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:07.151475 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:07.151722 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:07.151846 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:07.152222 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:07.152703 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:07.152846 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:07.152894 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 18:11:07.152963 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":133,"threads_started":1}
I20260810 18:11:07.160991 15592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:07.157013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.21.182.124:33035)
0810 18:11:07.157349 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:07.157352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:07.157369 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:11:07.157481 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:07.157483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:07.157602 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:07.157769 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:07.157928 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:07.158451 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:07.159320 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:07.159558 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:07.159641 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:07.160010 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:07.160470 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:07.160569 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:07.160661 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 18:11:07.160798 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":110,"threads_started":1}
I20260810 18:11:07.168913 15592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:07.164839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36730 (local address 127.21.182.124:33035)
0810 18:11:07.165011 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:07.165018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:07.165034 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:11:07.165083 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:07.165086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:07.165141 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:07.165244 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:07.165552 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:07.166269 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:07.167519 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:07.167930 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:07.168026 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:07.168389 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:07.168479 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:07.168527 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:07.168710 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 18:11:07.168769 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 18:11:07.169272 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:07.164784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.21.182.125:44205)
0810 18:11:07.164900 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:07.164904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:07.165283 (+ 379us) server_negotiation.cc:407] Connection header received
0810 18:11:07.165481 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:07.165487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:07.165561 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:07.165631 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:07.166034 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:07.166811 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:07.167957 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:07.168292 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:07.168399 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:07.168767 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:07.168847 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:07.168894 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:07.169091 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 18:11:07.169146 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 18:11:07.170150 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:07.164389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.21.182.126:38575)
0810 18:11:07.165134 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:07.165140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:07.165153 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:11:07.165263 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:07.165269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:07.165448 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:07.165568 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:07.165815 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:07.166668 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:07.167774 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:07.168357 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:07.168503 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:07.169257 (+ 754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:07.169636 (+ 379us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:07.169726 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:07.169925 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 18:11:07.169988 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":84,"threads_started":1}
I20260810 18:11:07.177282 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 15164
I20260810 18:11:07.187568 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 15252
I20260810 18:11:07.197902 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 15338
2026-08-10T18:11:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3355 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:11:07 dist-test-slave-27sr krb5kdc[15607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:11:07 dist-test-slave-27sr krb5kdc[15607](info): set up 2 sockets
Aug 10 18:11:07 dist-test-slave-27sr krb5kdc[15607](info): commencing operation
krb5kdc: starting...
W20260810 18:11:08.315300 22233 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/test-user.keytab.
Aug 10 18:11:08 dist-test-slave-27sr krb5kdc[15607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:11:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:11:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.182.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
Entry for principal kudu/127.21.182.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab.
I20260810 18:11:08.459599 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:42467
--webserver_interface=127.21.182.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:42625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:42467,127.21.182.125:33447,127.21.182.124:35621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:11:08.578903 15622 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:11:08.579300 15622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:08.579376 15622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:08.579459 15622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:08.583595 15622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:08.583709 15622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:08.583751 15622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:08.583793 15622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:08.583853 15622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:08.589452 15622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:42625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:42467,127.21.182.125:33447,127.21.182.124:35621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.126.keytab
--principal=kudu/127.21.182.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.126:42467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.182.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:08.590713 15622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:08.591858 15622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:08.601563 15630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:08.601610 15634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:08.601563 15631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:08.601998 15622 server_base.cc:1034] running on GCE node
I20260810 18:11:08.602617 15622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:08.603964 15622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:08.605163 15622 hybrid_clock.cc:648] HybridClock initialized: now 1786385468605126 us; error 50 us; skew 500 ppm
Aug 10 18:11:08 dist-test-slave-27sr krb5kdc[15607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385468, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:08.608352 15622 init.cc:377] Logged in from keytab as kudu/127.21.182.126@KRBTEST.COM (short username kudu)
I20260810 18:11:08.609321 15622 webserver.cc:460] Webserver started at http://127.21.182.126:32775/ using document root <none> and password file <none>
I20260810 18:11:08.609907 15622 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:08.609970 15622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:08.610153 15622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:08.612413 15622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/data/instance:
uuid: "32d220ee6b2c47279072299acde2c9b7"
format_stamp: "Formatted at 2026-08-10 18:11:08 on dist-test-slave-27sr"
server_key: "efc6077f4b598f7d43e1be632b1b6ad9"
server_key_iv: "0db7b6edaec274ecd8eaeb22684ad4ea"
server_key_version: "encryptionkey@0"
I20260810 18:11:08.612993 15622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/wal/instance:
uuid: "32d220ee6b2c47279072299acde2c9b7"
format_stamp: "Formatted at 2026-08-10 18:11:08 on dist-test-slave-27sr"
server_key: "efc6077f4b598f7d43e1be632b1b6ad9"
server_key_iv: "0db7b6edaec274ecd8eaeb22684ad4ea"
server_key_version: "encryptionkey@0"
I20260810 18:11:08.617061 15622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:11:08.619714 15642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:08.621133 15622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:11:08.621246 15622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/wal
uuid: "32d220ee6b2c47279072299acde2c9b7"
format_stamp: "Formatted at 2026-08-10 18:11:08 on dist-test-slave-27sr"
server_key: "efc6077f4b598f7d43e1be632b1b6ad9"
server_key_iv: "0db7b6edaec274ecd8eaeb22684ad4ea"
server_key_version: "encryptionkey@0"
I20260810 18:11:08.621345 15622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:08.656803 15622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:08.660048 15622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:08.660215 15622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:08.667642 15622 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.126:42467
I20260810 18:11:08.667979 15706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.126:42467 every 8 connection(s)
I20260810 18:11:08.668740 15622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/data/info.pb
I20260810 18:11:08.671712 15708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:08.675770 15708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:11:08.677753 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 15622
I20260810 18:11:08.677882 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-0/wal/instance
I20260810 18:11:08.678192 22233 external_mini_cluster.cc:1428] Setting key c5ec2d556173a55769cb9449013140f3
Aug 10 18:11:08 dist-test-slave-27sr krb5kdc[15607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385468, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.182.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.125@KRBTEST.COM" created.
I20260810 18:11:08.693570 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.678216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:42467 (local address 127.0.0.1:35000)
0810 18:11:08.678768 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.678824 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:08.679688 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:08.680598 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:08.680606 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:08.681188 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:08.681592 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:08.681613 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.682801 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.682807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:08.684161 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:08.684177 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.684997 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.685003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:08.685153 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:08.685915 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:08.685938 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:08.687788 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:08.690029 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:08.690040 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:08.690052 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:08.690391 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:08.690757 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:08.690762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:08.690764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:08.690890 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:08.691591 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:08.691598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:08.691755 (+ 157us) client_negotiation.cc:769] Sending connection context
0810 18:11:08.692019 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 18:11:08.692496 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":148,"thread_start_us":143,"threads_started":1}
I20260810 18:11:08.693567 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.678217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.21.182.126:42467)
0810 18:11:08.678877 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:08.678891 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:08.678941 (+ 50us) server_negotiation.cc:407] Connection header received
0810 18:11:08.680050 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:08.680059 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:08.680334 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:08.680555 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:08.681765 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.682650 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.684311 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.684804 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.684938 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:08.687963 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:08.687970 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:08.687981 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:08.688002 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:08.689863 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:08.690521 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:08.690527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:08.690529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:08.690651 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:08.690996 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:08.690999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:08.691001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:08.691461 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:08.691623 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:08.692089 (+ 466us) server_negotiation.cc:299] Negotiation successful
0810 18:11:08.692444 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":509,"spinlock_wait_cycles":11520,"thread_start_us":185,"threads_started":1}
I20260810 18:11:08.696246 15708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:11:08.697098 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.696813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:33447 (local address 127.0.0.1:40174)
0810 18:11:08.696947 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.697031 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 18:11:08.697371 15647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.182.125:33447: connect: Connection refused (error 111)
I20260810 18:11:08.698329 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.698103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:33447 (local address 127.0.0.1:40184)
0810 18:11:08.698214 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.698258 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 18:11:08.698570 15708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.125:33447: Network error: Client connection negotiation failed: client connection to 127.21.182.125:33447: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.125.keytab.
Entry for principal kudu/127.21.182.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.125.keytab.
I20260810 18:11:08.711134 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.125:33447
--webserver_interface=127.21.182.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.125.keytab
--principal=kudu/127.21.182.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:42625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:42467,127.21.182.125:33447,127.21.182.124:35621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:11:08.747846 15708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } attempt: 1
I20260810 18:11:08.748914 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.748574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:33447 (local address 127.0.0.1:40198)
0810 18:11:08.748741 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.748811 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 18:11:08.749688 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.749484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:33447 (local address 127.0.0.1:40206)
0810 18:11:08.749591 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.749636 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 18:11:08.749871 15708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.125:33447: Network error: Client connection negotiation failed: client connection to 127.21.182.125:33447: connect: Connection refused (error 111)
I20260810 18:11:08.818040 15708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } attempt: 2
I20260810 18:11:08.819016 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.818689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:33447 (local address 127.0.0.1:40222)
0810 18:11:08.818873 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.818938 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 18:11:08.819797 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.819594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:33447 (local address 127.0.0.1:40236)
0810 18:11:08.819700 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.819745 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.125:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 18:11:08.819950 15708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.125:33447: Network error: Client connection negotiation failed: client connection to 127.21.182.125:33447: connect: Connection refused (error 111)
W20260810 18:11:08.831660 15717 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:11:08.832050 15717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:08.832106 15717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:08.832208 15717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:08.836392 15717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:08.836454 15717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:08.836474 15717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:08.836491 15717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:08.836529 15717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:08.842093 15717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:42625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:42467,127.21.182.125:33447,127.21.182.124:35621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.125.keytab
--principal=kudu/127.21.182.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.125:33447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.182.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:08.843281 15717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:08.844452 15717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:08.853754 15725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:08.853884 15723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:08.853803 15717 server_base.cc:1034] running on GCE node
W20260810 18:11:08.853825 15722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:08.854700 15717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:08.855846 15717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:08.857054 15717 hybrid_clock.cc:648] HybridClock initialized: now 1786385468857019 us; error 48 us; skew 500 ppm
Aug 10 18:11:08 dist-test-slave-27sr krb5kdc[15607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385468, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:08.860008 15717 init.cc:377] Logged in from keytab as kudu/127.21.182.125@KRBTEST.COM (short username kudu)
I20260810 18:11:08.860930 15717 webserver.cc:460] Webserver started at http://127.21.182.125:40839/ using document root <none> and password file <none>
I20260810 18:11:08.861526 15717 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:08.861591 15717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:08.861848 15717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:08.864032 15717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/data/instance:
uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9"
format_stamp: "Formatted at 2026-08-10 18:11:08 on dist-test-slave-27sr"
server_key: "75a23c77dcecf1bf2c5b1ae3c622aa95"
server_key_iv: "ec155b0bdbfd073afd49e6f65e864a59"
server_key_version: "encryptionkey@0"
I20260810 18:11:08.864620 15717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/wal/instance:
uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9"
format_stamp: "Formatted at 2026-08-10 18:11:08 on dist-test-slave-27sr"
server_key: "75a23c77dcecf1bf2c5b1ae3c622aa95"
server_key_iv: "ec155b0bdbfd073afd49e6f65e864a59"
server_key_version: "encryptionkey@0"
I20260810 18:11:08.868654 15717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:11:08.871232 15734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:08.872562 15717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:11:08.872713 15717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/wal
uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9"
format_stamp: "Formatted at 2026-08-10 18:11:08 on dist-test-slave-27sr"
server_key: "75a23c77dcecf1bf2c5b1ae3c622aa95"
server_key_iv: "ec155b0bdbfd073afd49e6f65e864a59"
server_key_version: "encryptionkey@0"
I20260810 18:11:08.872846 15717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:08.882411 15717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:08.885588 15717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:08.885828 15717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:08.893870 15717 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.125:33447
I20260810 18:11:08.894192 15798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.125:33447 every 8 connection(s)
I20260810 18:11:08.895027 15717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/data/info.pb
I20260810 18:11:08.898623 15799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:08.898990 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 15717
I20260810 18:11:08.899129 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-1/wal/instance
I20260810 18:11:08.899530 22233 external_mini_cluster.cc:1428] Setting key 5f88165df6c6db95067130c9ec0880bf
I20260810 18:11:08.904054 15799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.182.124@KRBTEST.COM; defaulting to no policy
I20260810 18:11:08.911106 15708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.182.124@KRBTEST.COM" created.
Aug 10 18:11:08 dist-test-slave-27sr krb5kdc[15607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385468, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.125@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
Aug 10 18:11:08 dist-test-slave-27sr krb5kdc[15607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385468, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for kudu/127.21.182.125@KRBTEST.COM
I20260810 18:11:08.922879 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.906348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.21.182.126:42467)
0810 18:11:08.906505 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:08.906509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:08.907658 (+ 1149us) server_negotiation.cc:407] Connection header received
0810 18:11:08.908899 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:08.908903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:08.908979 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:08.909115 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:08.910476 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.911230 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.913155 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.913514 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.913597 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:08.916798 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:08.916805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:08.916811 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:08.916847 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:08.919306 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:08.920135 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:08.920140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:08.920142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:08.920246 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:08.920756 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:08.920761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:08.920765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:08.921040 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:08.921161 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:08.922215 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0810 18:11:08.922347 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:11:08.923611 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.906847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:42467 (local address 127.0.0.1:35012)
0810 18:11:08.907470 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.907522 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:08.908634 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:08.909302 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:08.909312 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:08.909775 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:08.910305 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:08.910321 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.911446 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.911453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:08.912986 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:08.912999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.913810 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.913816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:08.914140 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:08.914891 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:08.914908 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:08.916634 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:08.919469 (+ 2835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:08.919479 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:08.919503 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:08.919995 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:08.920369 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:08.920374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:08.920377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:08.920603 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:08.921178 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:08.921188 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:08.921559 (+ 371us) client_negotiation.cc:769] Sending connection context
0810 18:11:08.921841 (+ 282us) client_negotiation.cc:240] Negotiation successful
0810 18:11:08.922270 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":41,"thread_start_us":167,"threads_started":1}
I20260810 18:11:08.925482 15799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:11:08.925762 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.911642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:33447 (local address 127.0.0.1:40238)
0810 18:11:08.912050 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.912097 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:08.912422 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:08.913318 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:08.913325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:08.913570 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:08.913734 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:08.913743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.915765 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.915770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:08.916616 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:08.916628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.918022 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.918029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:08.918174 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:08.918779 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:08.918796 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:08.921279 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:08.923771 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:08.923774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:08.923776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:08.924081 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:08.924434 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:08.924438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:08.924440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:08.924551 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:08.925159 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:08.925163 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:08.925287 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 18:11:08.925403 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 18:11:08.925575 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260810 18:11:08.926100 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.912022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.21.182.125:33447)
0810 18:11:08.912737 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:08.912758 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:08.912787 (+ 29us) server_negotiation.cc:407] Connection header received
0810 18:11:08.912845 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:08.912851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:08.913087 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:08.913922 (+ 835us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:08.914191 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.915618 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.917193 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.917853 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.918027 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:08.921446 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:08.921460 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:08.921481 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:08.921515 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:08.923623 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:08.924234 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:08.924242 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:08.924245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:08.924335 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:08.924702 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:08.924705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:08.924706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:08.925036 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:08.925156 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:08.925541 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 18:11:08.925849 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":593,"thread_start_us":358,"threads_started":1}
I20260810 18:11:08.928118 15708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:11:08.928885 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.928641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45818)
0810 18:11:08.928753 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.928811 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 18:11:08.929651 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.929351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45826)
0810 18:11:08.929530 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.929586 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260810 18:11:08.929827 15708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:35621: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Aug 10 18:11:08 dist-test-slave-27sr krb5kdc[15607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385468, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.125@KRBTEST.COM for kudu/127.21.182.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.182.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.124.keytab.
Entry for principal kudu/127.21.182.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.124.keytab.
I20260810 18:11:08.936415 22233 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0GlVnd/build/debug/bin/kudu
/tmp/dist-test-task0GlVnd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.124:35621
--webserver_interface=127.21.182.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.124.keytab
--principal=kudu/127.21.182.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.182.84:42625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.182.126:42467,127.21.182.125:33447,127.21.182.124:35621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:11:08.938252 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.926055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:33447 (local address 127.0.0.1:40254)
0810 18:11:08.926159 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.926175 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:08.926331 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:08.927846 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:08.927849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:08.928074 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:08.928198 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:08.928208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.929332 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.929335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:08.930199 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:08.930211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.930707 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.930710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:08.930799 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:08.931341 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:08.931373 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:08.933186 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:08.936151 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:08.936158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:08.936163 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:08.936443 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:08.937086 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:08.937092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:08.937097 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:08.937208 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:08.937698 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:08.937701 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:08.937807 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 18:11:08.937938 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 18:11:08.938087 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 18:11:08.938452 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.926118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.21.182.125:33447)
0810 18:11:08.926761 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:08.926771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:08.926785 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:11:08.926822 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:08.926824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:08.926873 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:08.926977 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:08.928367 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.929217 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.930326 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:08.930588 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:08.930690 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:08.933360 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:08.933366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:08.933369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:08.933392 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:08.935992 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:08.936746 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:08.936751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:08.936753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:08.936933 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:08.937327 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:08.937333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:08.937337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:08.937598 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:08.937703 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:08.938019 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 18:11:08.938168 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":114,"threads_started":1}
I20260810 18:11:08.939232 15799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:11:08.940030 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.939716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45836)
0810 18:11:08.939840 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.939968 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 18:11:08.940353 15738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
I20260810 18:11:08.945549 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.941341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45850)
0810 18:11:08.945399 (+ 4058us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.945471 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3969}
W20260810 18:11:08.945726 15799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:35621: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
I20260810 18:11:08.961064 15708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } attempt: 1
I20260810 18:11:08.962111 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.961673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45856)
0810 18:11:08.961929 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.962009 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 18:11:08.962968 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.962698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45862)
0810 18:11:08.962843 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.962899 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 18:11:08.963120 15708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:35621: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
I20260810 18:11:08.995061 15799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } attempt: 1
I20260810 18:11:08.996172 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.995795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45866)
0810 18:11:08.996000 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.996082 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 18:11:08.997103 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:08.996848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45876)
0810 18:11:08.996976 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:08.997039 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 18:11:08.997316 15799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:35621: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
I20260810 18:11:09.038305 15708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } attempt: 2
I20260810 18:11:09.039352 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.038949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45888)
0810 18:11:09.039153 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:09.039242 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 18:11:09.040293 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.040013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45900)
0810 18:11:09.040142 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:09.040212 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 18:11:09.040454 15708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:35621: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
I20260810 18:11:09.065508 15799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } attempt: 2
I20260810 18:11:09.066680 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.066294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45912)
0810 18:11:09.066495 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:09.066574 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 18:11:09.067564 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.067294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45920)
0810 18:11:09.067430 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:09.067489 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 18:11:09.067741 15799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:35621: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
W20260810 18:11:09.072046 15806 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 18:11:09.072376 15806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:11:09.072402 15806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:11:09.072443 15806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:11:09.076776 15806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:11:09.076848 15806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:11:09.076869 15806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:11:09.076887 15806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:11:09.076903 15806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:11:09.082872 15806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.182.84:42625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.182.126:42467,127.21.182.125:33447,127.21.182.124:35621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/krb5kdc/kudu_127.21.182.124.keytab
--principal=kudu/127.21.182.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.182.124:35621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.182.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 17:11:04 UTC on ccycloud.quasar-msnrfz.root.comops.site
build id 4578357
I20260810 18:11:09.084127 15806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:11:09.085238 15806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:11:09.094822 15820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:09.094913 15817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:11:09.094897 15816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:11:09.095100 15806 server_base.cc:1034] running on GCE node
I20260810 18:11:09.095849 15806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:11:09.097254 15806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:11:09.098434 15806 hybrid_clock.cc:648] HybridClock initialized: now 1786385469098419 us; error 33 us; skew 500 ppm
Aug 10 18:11:09 dist-test-slave-27sr krb5kdc[15607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385469, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:11:09.101935 15806 init.cc:377] Logged in from keytab as kudu/127.21.182.124@KRBTEST.COM (short username kudu)
I20260810 18:11:09.103041 15806 webserver.cc:460] Webserver started at http://127.21.182.124:33507/ using document root <none> and password file <none>
I20260810 18:11:09.103672 15806 fs_manager.cc:359] Metadata directory not provided
I20260810 18:11:09.103771 15806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:11:09.104096 15806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:11:09.106385 15806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/data/instance:
uuid: "167843c9356548a69702f1aed30404ab"
format_stamp: "Formatted at 2026-08-10 18:11:09 on dist-test-slave-27sr"
server_key: "ea0eb8e74f2c276ced75f35272a4ac6e"
server_key_iv: "b46772c46df5ac720c2ce2d36be44b53"
server_key_version: "encryptionkey@0"
I20260810 18:11:09.107054 15806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/wal/instance:
uuid: "167843c9356548a69702f1aed30404ab"
format_stamp: "Formatted at 2026-08-10 18:11:09 on dist-test-slave-27sr"
server_key: "ea0eb8e74f2c276ced75f35272a4ac6e"
server_key_iv: "b46772c46df5ac720c2ce2d36be44b53"
server_key_version: "encryptionkey@0"
I20260810 18:11:09.111284 15806 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:11:09.114184 15827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:11:09.115562 15806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:11:09.115737 15806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/data,/tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/wal
uuid: "167843c9356548a69702f1aed30404ab"
format_stamp: "Formatted at 2026-08-10 18:11:09 on dist-test-slave-27sr"
server_key: "ea0eb8e74f2c276ced75f35272a4ac6e"
server_key_iv: "b46772c46df5ac720c2ce2d36be44b53"
server_key_version: "encryptionkey@0"
I20260810 18:11:09.115880 15806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:11:09.139655 15708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } attempt: 3
I20260810 18:11:09.140654 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.140301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45926)
0810 18:11:09.140485 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:09.140559 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 18:11:09.141366 15806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:11:09.141471 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.141224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45936)
0810 18:11:09.141350 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:09.141408 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 18:11:09.141659 15708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.182.124:35621: Network error: Client connection negotiation failed: client connection to 127.21.182.124:35621: connect: Connection refused (error 111)
I20260810 18:11:09.144721 15806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:11:09.144899 15806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:11:09.152473 15806 rpc_server.cc:307] RPC server started. Bound to: 127.21.182.124:35621
I20260810 18:11:09.152833 15884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.182.124:35621 every 8 connection(s)
I20260810 18:11:09.153622 15806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/data/info.pb
I20260810 18:11:09.156788 15885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:11:09.158910 15799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } attempt: 3
I20260810 18:11:09.161700 15885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:11:09.163112 22233 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0GlVnd/build/debug/bin/kudu as pid 15806
I20260810 18:11:09.163252 22233 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0GlVnd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786385164390354-22233-0/minicluster-data/master-2/wal/instance
I20260810 18:11:09.163599 22233 external_mini_cluster.cc:1428] Setting key c02492cd65060d46c75fd978588e8644
Aug 10 18:11:09 dist-test-slave-27sr krb5kdc[15607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385468, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.125@KRBTEST.COM for kudu/127.21.182.124@KRBTEST.COM
Aug 10 18:11:09 dist-test-slave-27sr krb5kdc[15607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385469, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.124@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
Aug 10 18:11:09 dist-test-slave-27sr krb5kdc[15607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.124@KRBTEST.COM
I20260810 18:11:09.173727 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.159553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45952)
0810 18:11:09.159726 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:09.159744 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:09.159853 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:09.162530 (+ 2677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:09.162534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:09.162750 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:09.162879 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:09.162888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.164791 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.164796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:09.165624 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:09.165638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.166614 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.166618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:09.166712 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.167321 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:09.167343 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:09.169029 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:09.171366 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:09.171370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:09.171372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:09.171594 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:09.172037 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:09.172040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:09.172042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:09.172142 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:09.173111 (+ 969us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:09.173116 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:09.173249 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 18:11:09.173352 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 18:11:09.173526 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 18:11:09.174969 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.164167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.21.182.126:42467)
0810 18:11:09.164286 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.164290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.164728 (+ 438us) server_negotiation.cc:407] Connection header received
0810 18:11:09.164856 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.164859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.164920 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.165015 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.165859 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.166436 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.167622 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.167851 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.168008 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.171002 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:09.171009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:09.171015 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:09.171053 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:09.172761 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.173322 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.173326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.173328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.173417 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.173980 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.173984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.173986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.174217 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:09.174278 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:09.174678 (+ 400us) server_negotiation.cc:299] Negotiation successful
0810 18:11:09.174820 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:11:09.175815 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.160463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.21.182.124:35621)
0810 18:11:09.160963 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.160979 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.161017 (+ 38us) server_negotiation.cc:407] Connection header received
0810 18:11:09.161804 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.161825 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.162187 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.162538 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.163050 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.164604 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.165770 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.166478 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.166695 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.169190 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:09.169198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:09.169213 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:09.169239 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:09.171253 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.171720 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.171726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.171733 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.171891 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.172274 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.172277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.172279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.172953 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:09.174008 (+ 1055us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:09.174201 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 18:11:09.174591 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":345,"thread_start_us":161,"threads_started":1}
I20260810 18:11:09.175815 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.163845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.126:42467 (local address 127.0.0.1:35026)
0810 18:11:09.164568 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:09.164621 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:09.164752 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:09.165101 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:09.165107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:09.165562 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:09.165718 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:09.165728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.166573 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.166577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:09.167485 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:09.167496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.168054 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.168058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:09.168171 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.168840 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:09.168862 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:09.170827 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:09.172892 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:09.172903 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:09.172910 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:09.173209 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:09.173721 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:09.173723 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:09.173725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:09.173849 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:09.174361 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:09.174366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:09.174470 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 18:11:09.174616 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 18:11:09.174812 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":33,"spinlock_wait_cycles":1792,"thread_start_us":142,"threads_started":1}
I20260810 18:11:09.177704 15885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:11:09.178864 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.164609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.21.182.124:35621)
0810 18:11:09.165000 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.165008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.165188 (+ 180us) server_negotiation.cc:407] Connection header received
0810 18:11:09.165223 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.165225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.165265 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.165359 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.166260 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.166993 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.168416 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.168730 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.168840 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.171942 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:09.171948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:09.171954 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:09.171984 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:09.174598 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.175113 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.175116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.175118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.175204 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.175531 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.175535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.175537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.175740 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:09.176334 (+ 594us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:09.178506 (+ 2172us) server_negotiation.cc:299] Negotiation successful
0810 18:11:09.178662 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":121,"threads_started":1}
I20260810 18:11:09.181058 15799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9: Bootstrap starting.
Aug 10 18:11:09 dist-test-slave-27sr krb5kdc[15607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385469, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.124@KRBTEST.COM for kudu/127.21.182.125@KRBTEST.COM
I20260810 18:11:09.184139 15799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:09.185201 15799 log.cc:824] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:09.187646 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.178313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.125:33447 (local address 127.0.0.1:40266)
0810 18:11:09.178478 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:09.178492 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:09.178567 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:09.178853 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:09.178855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:09.179142 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:09.179258 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:09.179265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.180273 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.180276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:09.181029 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:09.181041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.181574 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.181578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:09.181672 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.182137 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:09.182151 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:09.183532 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:09.185856 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:09.185860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:09.185862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:09.186138 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:09.186509 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:09.186513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:09.186515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:09.186612 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:09.187118 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:09.187121 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:09.187227 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 18:11:09.187357 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 18:11:09.187509 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 18:11:09.187829 15799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9: No bootstrap required, opened a new log
I20260810 18:11:09.188135 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.178418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.21.182.125:33447)
0810 18:11:09.178543 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.178548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.178562 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:11:09.178666 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.178669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.178718 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.178811 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.179409 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.180150 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.181188 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.181461 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.181545 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.183663 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:09.183667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:09.183670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:09.183695 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:09.185722 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.186291 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.186298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.186303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.186396 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.186710 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.186713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.186715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.187008 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:09.187120 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:09.187795 (+ 675us) server_negotiation.cc:299] Negotiation successful
0810 18:11:09.188000 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:11:09.188937 15885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:11:09 dist-test-slave-27sr krb5kdc[15607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.126@KRBTEST.COM
I20260810 18:11:09.192529 15799 raft_consensus.cc:348] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } }
I20260810 18:11:09.192847 15799 raft_consensus.cc:729] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da5ea188e6d744b7aabd9fb2abc9cbb9, State: Initialized, Role: FOLLOWER
I20260810 18:11:09.193246 15799 consensus_queue.cc:260] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } }
I20260810 18:11:09.194034 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.180424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.21.182.126:42467)
0810 18:11:09.184098 (+ 3674us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.184104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.184120 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:11:09.184168 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.184173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.184230 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.184325 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.185237 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.185866 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.186963 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.187280 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.187394 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.190404 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:09.190410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:09.190412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:09.190444 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:09.192042 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.192570 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.192574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.192575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.192684 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.192988 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.192992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.192994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.193233 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:09.193640 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:09.193730 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 18:11:09.193879 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3544}
I20260810 18:11:09.194259 15893 sys_catalog.cc:455] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } } }
I20260810 18:11:09.194427 15893 sys_catalog.cc:458] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:09.195061 15799 sys_catalog.cc:564] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9 [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 18:11:09 dist-test-slave-27sr krb5kdc[15607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385469, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.124@KRBTEST.COM for kudu/127.21.182.124@KRBTEST.COM
I20260810 18:11:09.201097 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.189519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45970)
0810 18:11:09.189688 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:09.189701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:09.189874 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:09.190186 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:09.190188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:09.190365 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:09.190466 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:09.190473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.192200 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.192203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:09.192965 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:09.192977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.193592 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.193595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:09.193668 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.194173 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:09.194188 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:09.195670 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:09.198086 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:09.198089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:09.198092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:09.198340 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:09.198627 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:09.198629 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:09.198631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:09.198742 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:09.200528 (+ 1786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:09.200532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:09.200649 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 18:11:09.200765 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 18:11:09.200919 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
W20260810 18:11:09.201596 15905 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:11:09.201671 15905 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:11:09.201704 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.189614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.21.182.124:35621)
0810 18:11:09.189743 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.189748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.189766 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:11:09.189998 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.190005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.190067 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.190158 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.190588 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.191367 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.193088 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.193374 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.193474 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.195838 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:09.195845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:09.195851 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:09.195882 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:09.197729 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.198447 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.198452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.198456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.198542 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.198845 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.198848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.198849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.200387 (+ 1538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:09.200492 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:09.201432 (+ 940us) server_negotiation.cc:299] Negotiation successful
0810 18:11:09.201551 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:11:09.200742 15717 master_runner.cc:186] Error getting master registration for 127.21.182.126:42467: OK, instance_id { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" instance_seqno: 1786385468665559 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:11:09.203735 15717 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 10 18:11:09 dist-test-slave-27sr krb5kdc[15607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.182.125@KRBTEST.COM
I20260810 18:11:09.207201 15885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab: Bootstrap starting.
I20260810 18:11:09.209370 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.196050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.21.182.125:33447)
0810 18:11:09.196516 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.196523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.196541 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:11:09.196589 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.196592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.196647 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.196742 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.197624 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.198391 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.199567 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.199926 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.200030 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.205394 (+ 5364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:09.205400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:09.205403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:09.205439 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:09.207316 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.207874 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.207878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.207880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.208005 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.208337 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.208340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.208342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.208606 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:09.208833 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:09.209048 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 18:11:09.209194 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:11:09.210183 15885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:09.211073 15885 log.cc:824] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:09.213680 15885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab: No bootstrap required, opened a new log
I20260810 18:11:09.214314 22233 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:11:09.215816 15678 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "32d220ee6b2c47279072299acde2c9b7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35030
I20260810 18:11:09.217742 15885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } }
I20260810 18:11:09.218021 15885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 167843c9356548a69702f1aed30404ab, State: Initialized, Role: FOLLOWER
I20260810 18:11:09.218386 15885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } }
I20260810 18:11:09.218945 15910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } } }
I20260810 18:11:09.219133 15910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:09.219970 15885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab [sys.catalog]: configured and running, proceeding with master startup.
W20260810 18:11:09.226284 15924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:11:09.226379 15924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:11:09.224376 15806 master_runner.cc:186] Error getting master registration for 127.21.182.126:42467: OK, instance_id { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" instance_seqno: 1786385468665559 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 18:11:09.227778 15806 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 18:11:09.305845 15708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } attempt: 4
Aug 10 18:11:09 dist-test-slave-27sr krb5kdc[15607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786385468, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.182.126@KRBTEST.COM for kudu/127.21.182.124@KRBTEST.COM
I20260810 18:11:09.317134 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.306665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.21.182.124:35621)
0810 18:11:09.306819 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.306824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.306903 (+ 79us) server_negotiation.cc:407] Connection header received
0810 18:11:09.307071 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.307074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.307134 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.307239 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.307811 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.308636 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.309844 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.310200 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.310302 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.313255 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:09.313260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:09.313263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:09.313301 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:09.315128 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.315669 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.315673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.315675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.315779 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.316144 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.316147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.316149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.316400 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:09.316506 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:09.316816 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 18:11:09.316953 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:11:09.317149 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.306559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.182.124:35621 (local address 127.0.0.1:45972)
0810 18:11:09.306815 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 18:11:09.306832 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:11:09.306945 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:11:09.307296 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:11:09.307299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:11:09.307540 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:11:09.307668 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:09.307676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.308821 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.308825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:09.309673 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:11:09.309685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.310457 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.310461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:11:09.310558 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.311166 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:11:09.311182 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:11:09.313064 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:11:09.315272 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:09.315277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:09.315279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:09.315561 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:09.315884 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:11:09.315887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:11:09.315917 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:11:09.316031 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:11:09.316513 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:11:09.316516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:11:09.316634 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 18:11:09.316751 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 18:11:09.316967 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260810 18:11:09.321394 15708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7: Bootstrap starting.
I20260810 18:11:09.325047 15708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7: Neither blocks nor log segments found. Creating new log.
I20260810 18:11:09.326135 15708 log.cc:824] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7: Log is configured to *not* fsync() on all Append() calls
I20260810 18:11:09.328938 15708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7: No bootstrap required, opened a new log
I20260810 18:11:09.332410 15708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } }
I20260810 18:11:09.332700 15708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d220ee6b2c47279072299acde2c9b7, State: Initialized, Role: FOLLOWER
I20260810 18:11:09.333089 15708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } }
I20260810 18:11:09.334697 15708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:11:09.334831 15678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 18:11:09.334977 15678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:09.336525 15678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } }
I20260810 18:11:09.337352 15929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } } }
I20260810 18:11:09.338081 15929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:09.338402 15768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32d220ee6b2c47279072299acde2c9b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9"
I20260810 18:11:09.338697 15768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:09.340257 15768 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32d220ee6b2c47279072299acde2c9b7 in term 1.
I20260810 18:11:09.341207 15647 leader_election.cc:302] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d220ee6b2c47279072299acde2c9b7, da5ea188e6d744b7aabd9fb2abc9cbb9; no voters:
I20260810 18:11:09.342344 15678 leader_election.cc:288] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [CANDIDATE]: Term 1 election: Requested vote from peers da5ea188e6d744b7aabd9fb2abc9cbb9 (127.21.182.125:33447), 167843c9356548a69702f1aed30404ab (127.21.182.124:35621)
I20260810 18:11:09.342777 15857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32d220ee6b2c47279072299acde2c9b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "167843c9356548a69702f1aed30404ab"
I20260810 18:11:09.343075 15857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:11:09.344534 15857 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32d220ee6b2c47279072299acde2c9b7 in term 1.
I20260810 18:11:09.344787 15929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:11:09.345049 15929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [term 1 LEADER]: Becoming Leader. State: Replica: 32d220ee6b2c47279072299acde2c9b7, State: Running, Role: LEADER
I20260810 18:11:09.345475 15929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } }
I20260810 18:11:09.347664 15929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32d220ee6b2c47279072299acde2c9b7. Latest consensus state: current_term: 1 leader_uuid: "32d220ee6b2c47279072299acde2c9b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } } }
I20260810 18:11:09.347821 15929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:09.349771 15622 master_runner.cc:263] No leader master found from master 32d220ee6b2c47279072299acde2c9b7
I20260810 18:11:09.349843 15622 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 18:11:09.349638 15942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:11:09.349915 15942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:11:09.352202 15952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:11:09.353410 15952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:11:09.361939 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.349336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.21.182.125:33447)
0810 18:11:09.349498 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.349503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.351832 (+ 2329us) server_negotiation.cc:407] Connection header received
0810 18:11:09.351929 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.351933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.351997 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.352084 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.353095 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.353942 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.354964 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.355314 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.355424 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.357775 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:09.357779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:09.357781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:09.357814 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:09.359725 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.360381 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.360385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.360387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.360487 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.360856 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.360860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.360861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.361105 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:09.361216 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:09.361610 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 18:11:09.361753 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 18:11:09.362133 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.348347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.21.182.126:42467)
0810 18:11:09.348481 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.348486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.350251 (+ 1765us) server_negotiation.cc:407] Connection header received
0810 18:11:09.350365 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.350367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.350422 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.350516 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.351483 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.352302 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.354990 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.355307 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.355387 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.357578 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:09.357583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:09.357586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:09.357617 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:09.359640 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.360320 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.360324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.360326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.360431 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.360767 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.360771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.360773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.361037 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:09.361146 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:09.361819 (+ 673us) server_negotiation.cc:299] Negotiation successful
0810 18:11:09.361949 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:11:09.368135 15768 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9 [term 1 FOLLOWER]: Refusing update from remote peer 32d220ee6b2c47279072299acde2c9b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:09.368135 15857 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab [term 1 FOLLOWER]: Refusing update from remote peer 32d220ee6b2c47279072299acde2c9b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:11:09.369098 15929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:09.369572 15929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:11:09.371773 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:09.349738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.21.182.124:35621)
0810 18:11:09.349877 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.349882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.356211 (+ 6329us) server_negotiation.cc:407] Connection header received
0810 18:11:09.356453 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.356457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.356532 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.356596 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.363133 (+ 6537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.363726 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.364711 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:09.365058 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:09.365172 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:09.367219 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:11:09.367223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:11:09.367226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:11:09.367259 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:11:09.369365 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.370165 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.370169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.370171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.370268 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:11:09.370600 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:11:09.370604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:11:09.370605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:11:09.370926 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:11:09.371052 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:09.371448 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 18:11:09.371585 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:11:09.378628 15910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32d220ee6b2c47279072299acde2c9b7. Latest consensus state: current_term: 1 leader_uuid: "32d220ee6b2c47279072299acde2c9b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } } }
I20260810 18:11:09.378872 15910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:09.380352 15893 sys_catalog.cc:455] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32d220ee6b2c47279072299acde2c9b7. Latest consensus state: current_term: 1 leader_uuid: "32d220ee6b2c47279072299acde2c9b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } } }
I20260810 18:11:09.380505 15893 sys_catalog.cc:458] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:09.382860 15950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32d220ee6b2c47279072299acde2c9b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } } }
I20260810 18:11:09.383030 15950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:09.383774 15956 mvcc.cc:205] Tried to move back new op lower bound from 7317034882521858048 to 7317034882443350016. Current Snapshot: MvccSnapshot[applied={T|T < 7317034882521858048}]
I20260810 18:11:09.385002 15910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32d220ee6b2c47279072299acde2c9b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } } }
I20260810 18:11:09.385143 15910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab [sys.catalog]: This master's current role is: FOLLOWER
W20260810 18:11:09.387558 15740 tablet.cc:2369] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:11:09.388765 15929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32d220ee6b2c47279072299acde2c9b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } } }
I20260810 18:11:09.388926 15929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7 [sys.catalog]: This master's current role is: LEADER
I20260810 18:11:09.390232 15893 sys_catalog.cc:455] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32d220ee6b2c47279072299acde2c9b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d220ee6b2c47279072299acde2c9b7" member_type: VOTER last_known_addr { host: "127.21.182.126" port: 42467 } } peers { permanent_uuid: "da5ea188e6d744b7aabd9fb2abc9cbb9" member_type: VOTER last_known_addr { host: "127.21.182.125" port: 33447 } } peers { permanent_uuid: "167843c9356548a69702f1aed30404ab" member_type: VOTER last_known_addr { host: "127.21.182.124" port: 35621 } } }
I20260810 18:11:09.390376 15893 sys_catalog.cc:458] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:11:09.391258 15958 mvcc.cc:205] Tried to move back new op lower bound from 7317034882521858048 to 7317034882443350016. Current Snapshot: MvccSnapshot[applied={T|T < 7317034882521858048}]
I20260810 18:11:09.392292 15960 mvcc.cc:205] Tried to move back new op lower bound from 7317034882521858048 to 7317034882443350016. Current Snapshot: MvccSnapshot[applied={T|T < 7317034882521858048}]
I20260810 18:11:09.393997 15952 catalog_manager.cc:1347] Generated new cluster ID: cf89971ffc1f46ed9e3f475db6311064
I20260810 18:11:09.394081 15952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:11:09.414079 15952 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:11:09.415530 15952 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:11:09.449321 15952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32d220ee6b2c47279072299acde2c9b7: Generated new TSK 0
I20260810 18:11:09.450274 15952 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:11:09.475651 15805 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:09.474095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.21.182.125:33447)
0810 18:11:09.474221 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.474226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.474596 (+ 370us) server_negotiation.cc:407] Connection header received
0810 18:11:09.474665 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.474668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.474730 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.474829 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.475490 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40278: BlockingRecv error: recv got EOF from 127.0.0.1:40278 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 18:11:09.475673 15891 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:09.473693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.21.182.124:35621)
0810 18:11:09.473831 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.473836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.474539 (+ 703us) server_negotiation.cc:407] Connection header received
0810 18:11:09.474714 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.474717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.474767 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.474867 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.475562 (+ 695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45980: BlockingRecv error: recv got EOF from 127.0.0.1:45980 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 18:11:09.476538 15711 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:09.474095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.21.182.126:42467)
0810 18:11:09.474221 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.474226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.475111 (+ 885us) server_negotiation.cc:407] Connection header received
0810 18:11:09.475305 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.475309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.475362 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.475414 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.476377 (+ 963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35044: BlockingRecv error: recv got EOF from 127.0.0.1:35044 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 18:11:09.479708 15805 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:09.478598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.21.182.125:33447)
0810 18:11:09.478732 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.478736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.478873 (+ 137us) server_negotiation.cc:407] Connection header received
0810 18:11:09.479012 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.479015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.479102 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.479190 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.479590 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40280: BlockingRecv error: recv got EOF from 127.0.0.1:40280 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 18:11:09.483978 15805 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:09.482990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40282 (local address 127.21.182.125:33447)
0810 18:11:09.483112 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.483116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.483127 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:11:09.483171 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.483173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.483216 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.483280 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.483775 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40282: BlockingRecv error: recv got EOF from 127.0.0.1:40282 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 18:11:09.488972 15805 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:09.487830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.21.182.125:33447)
0810 18:11:09.488024 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.488028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.488173 (+ 145us) server_negotiation.cc:407] Connection header received
0810 18:11:09.488295 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.488298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.488355 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.488448 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.488852 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40288: BlockingRecv error: recv got EOF from 127.0.0.1:40288 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
W20260810 18:11:09.496428 15805 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:09.495074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.21.182.125:33447)
0810 18:11:09.495285 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.495290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.495380 (+ 90us) server_negotiation.cc:407] Connection header received
0810 18:11:09.495514 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.495518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.495581 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.495722 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.496302 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40292: BlockingRecv error: recv got EOF from 127.0.0.1:40292 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260810 18:11:09.507863 15805 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:09.506596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.21.182.125:33447)
0810 18:11:09.506755 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.506760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.506881 (+ 121us) server_negotiation.cc:407] Connection header received
0810 18:11:09.507019 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.507022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.507093 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.507197 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.507728 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40302: BlockingRecv error: recv got EOF from 127.0.0.1:40302 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 18:11:09.527431 15805 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:09.526201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.21.182.125:33447)
0810 18:11:09.526368 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.526373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.526502 (+ 129us) server_negotiation.cc:407] Connection header received
0810 18:11:09.526621 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.526625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.526695 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.526806 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.527294 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40308: BlockingRecv error: recv got EOF from 127.0.0.1:40308 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260810 18:11:09.562836 15805 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:09.561623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.21.182.125:33447)
0810 18:11:09.561843 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.561847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.561920 (+ 73us) server_negotiation.cc:407] Connection header received
0810 18:11:09.562053 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.562056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.562109 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.562188 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.562671 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40316: BlockingRecv error: recv got EOF from 127.0.0.1:40316 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260810 18:11:09.630542 15805 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:09.629183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.21.182.125:33447)
0810 18:11:09.629393 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.629398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.629581 (+ 183us) server_negotiation.cc:407] Connection header received
0810 18:11:09.629772 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.629775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.629829 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.629932 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.630428 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40326: BlockingRecv error: recv got EOF from 127.0.0.1:40326 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 18:11:09.762563 15805 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:09.761137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40328 (local address 127.21.182.125:33447)
0810 18:11:09.761362 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:09.761368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:09.761520 (+ 152us) server_negotiation.cc:407] Connection header received
0810 18:11:09.761716 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:09.761725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:09.761777 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:09.761909 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:09.762414 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40328: BlockingRecv error: recv got EOF from 127.0.0.1:40328 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260810 18:11:10.022430 15805 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:11:10.021200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40332 (local address 127.21.182.125:33447)
0810 18:11:10.021382 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:10.021386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:10.021407 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:11:10.021578 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:10.021581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:10.021636 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:10.021714 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:11:10.022289 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40332: BlockingRecv error: recv got EOF from 127.0.0.1:40332 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 18:11:10.203264 15905 catalog_manager.cc:1259] Loaded cluster ID: cf89971ffc1f46ed9e3f475db6311064
I20260810 18:11:10.203357 15905 catalog_manager.cc:1552] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9: loading cluster ID for follower catalog manager: success
I20260810 18:11:10.205214 15905 catalog_manager.cc:1574] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9: acquiring CA information for follower catalog manager: success
I20260810 18:11:10.206521 15905 catalog_manager.cc:1602] T 00000000000000000000000000000000 P da5ea188e6d744b7aabd9fb2abc9cbb9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:11:10.227851 15924 catalog_manager.cc:1259] Loaded cluster ID: cf89971ffc1f46ed9e3f475db6311064
I20260810 18:11:10.228026 15924 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab: loading cluster ID for follower catalog manager: success
I20260810 18:11:10.229725 15924 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab: acquiring CA information for follower catalog manager: success
I20260810 18:11:10.230917 15924 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 167843c9356548a69702f1aed30404ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:11:10.541373 16040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:10.536963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.21.182.125:33447)
0810 18:11:10.537230 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:10.537234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:10.537278 (+ 44us) server_negotiation.cc:407] Connection header received
0810 18:11:10.537449 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:10.537452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:10.537598 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:10.537737 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:10.537989 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:10.538589 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:10.539587 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:10.539848 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:10.540004 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:10.540380 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:10.540928 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:10.541037 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:10.541169 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 18:11:10.541234 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":95,"threads_started":1}
I20260810 18:11:10.549016 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:10.544882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.21.182.124:35621)
0810 18:11:10.545104 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:10.545108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:10.545122 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:11:10.545286 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:10.545288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:10.545410 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:10.545515 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:10.545710 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:10.546232 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:10.547302 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:10.547550 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:10.547670 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:10.548100 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:10.548580 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:10.548681 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:10.548832 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 18:11:10.548889 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
I20260810 18:11:10.556789 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:10.552932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45986 (local address 127.21.182.124:35621)
0810 18:11:10.553044 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:10.553048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:10.553257 (+ 209us) server_negotiation.cc:407] Connection header received
0810 18:11:10.553313 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:10.553316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:10.553372 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:10.553445 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:10.553729 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:10.554369 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:10.555514 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:10.555867 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:10.555993 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:10.556255 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:10.556353 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:10.556418 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:10.556584 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 18:11:10.556643 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 18:11:10.557185 16040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:10.552646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40358 (local address 127.21.182.125:33447)
0810 18:11:10.552802 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:10.552807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:10.553210 (+ 403us) server_negotiation.cc:407] Connection header received
0810 18:11:10.553451 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:10.553455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:10.553518 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:10.553599 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:10.553892 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:10.554709 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:10.555946 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:10.556262 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:10.556377 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:10.556670 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:10.556778 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:10.556830 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:10.557004 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 18:11:10.557063 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:11:10.557852 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:11:10.552928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.21.182.126:42467)
0810 18:11:10.553231 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0810 18:11:10.553238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:11:10.553253 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:11:10.553360 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:11:10.553363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:11:10.553541 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:11:10.553695 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:11:10.554028 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:10.554839 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:10.555876 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:11:10.556292 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:11:10.556417 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:11:10.556943 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:11:10.557356 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:11:10.557426 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 18:11:10.557597 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 18:11:10.557665 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
I20260810 18:11:10.561326 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 15622
I20260810 18:11:10.573340 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 15717
I20260810 18:11:10.583282 22233 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0GlVnd/build/debug/bin/kudu with pid 15806
2026-08-10T18:11:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3388 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6744 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (306037 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST